CanYouPutPHIinanLLM?WhataBAAActuallyCovers

Introduction

Your AI feature works. Somebody on the clinical side loves it. Then a customer sends a security questionnaire asking which sub-processors can see patient data, and you realise you can't answer it, because nobody ever traced where the prompt goes after it leaves your server.

Or the worse version. It already shipped, and the question arrived afterwards.

Search for help and you'll get the same page written forty different ways. Standard ChatGPT is not HIPAA compliant. True, and useless, because you knew that already. What almost nobody writes down is the part you actually need: which vendors will sign, what the signature reaches, and what still has to be true in your own code afterwards.

This guide covers that second half. Every vendor claim below was checked against current published terms on 26 August 2026, and where sources contradict each other we say so instead of picking the confident-sounding one. For the related budget question, which drivers an AI feature adds to a healthcare build, see our healthcare app development cost breakdown.

> A BAA is a contract about who is liable when something goes wrong. It is not a control, and it has never once stopped a single byte from going anywhere. Teams keep treating the signature as the finish line when it is closer to the starting gun.

Is ChatGPT HIPAA Compliant?

The consumer product isn't. ChatGPT Free and Plus carry no Business Associate Agreement, and pasting patient information into either is a disclosure to a vendor you have no agreement with. That much is settled and nobody credible argues otherwise.

The API is a different product and the answer changes. OpenAI will review a BAA request, sent to their published BAA contact address, on a case by case basis, and where it is granted the coverage is tied to endpoints configured for zero data retention. They have since introduced a dedicated healthcare offering as well. So the honest answer to the head question is that it depends entirely on which OpenAI you mean, and the phrase "is ChatGPT HIPAA compliant" hides four different products.

Here's where we stop being precise, deliberately. Published sources disagree about the workspace and enterprise tiers. Some say flatly that OpenAI won't sign for Enterprise. Others say qualifying enterprise healthcare customers are supported. Both were written by people with no obvious reason to lie, which usually means the product moved underneath them and one of the two is stale.

We're not going to resolve that for you, because we'd be guessing and you'd be the one relying on it. Get the answer for your exact surface from the vendor in writing. That sentence is worth more than a confident paragraph, and any page that hands you a crisp answer to a question this volatile is telling you something about how carefully it was written.

The reason this matters beyond pedantry is that procurement and engineering usually ask different questions here. Legal asks whether the vendor signs. Engineering asks whether this specific call, on this specific endpoint, with this specific feature enabled, is inside the agreement. Only the second question has ever protected anyone.

Which AI Vendors Will Actually Sign a BAA?

All the major ones, as of 2026. The differences that matter are not whether they sign, but which tiers, endpoints and features the signature reaches.

The row most teams should read twice is AWS Bedrock. If you already run on AWS you very likely already have a Business Associate Addendum in place, and Bedrock being HIPAA-eligible means one agreement you already signed covers access to several model providers at once. That is a procurement shortcut worth weeks, and we have never seen it framed that way on a page selling compliance software.

It also changes the vendor-selection conversation. Teams agonise over which model to standardise on partly because each new provider means a new agreement and a new sub-processor disclosure. Route through a provider you already have covered and that cost mostly disappears, which frees you to pick the model on merit instead of on paperwork.

One caution that AWS states plainly and people still miss. HIPAA-eligible means the service may be used with protected health information under a BAA. It does not mean your workload is compliant. The eligibility is about the service. The compliance is about what you did with it.

What Does a BAA With an AI Vendor Actually Cover?

It covers liability and obligations. It commits the vendor to apply safeguards to protected health information, to restrict how they use and disclose it, to report breaches to you, and to hold their own subcontractors to the same terms. That's genuinely valuable and you can't operate without it.

What it doesn't do is change what your code does. This is the gap that swallows teams whole.

A BAA is a contract. Contracts allocate consequences after the fact. They don't sit in the request path and refuse to forward a prompt. If your application sends patient data somewhere the agreement doesn't reach, the agreement doesn't know, doesn't care, and won't help you, because the thing you signed was about a different destination entirely.

The failure mode is boringly consistent and we've watched it play out more than once. Legal secures the BAA with the model provider. Engineering integrates the model provider. Both teams consider the compliance work complete, because from where each of them is standing it looks finished. Nobody enumerates the other seven places request content lands on its way through the system.

What it also doesn't cover: your own people. A BAA with OpenAI says nothing about a support engineer pasting a failing prompt into a ticketing system to debug it, or a product manager exporting a hundred real conversations into a spreadsheet to review output quality. Those are disclosures too. They're also the ones your architecture diagram will never show you, because they happen in the gaps between systems, where humans do their jobs.

Why Does Zero Data Retention Decide Whether Your BAA Means Anything?

Zero data retention means the provider does not store your request and response after serving it. No copy for abuse review, no copy for debugging, nothing that persists past the call.

Standard API behaviour is usually the opposite. Providers commonly retain requests for a window, often around 30 days, so they can investigate misuse. For ordinary workloads that is a sensible default nobody thinks about. For protected health information it means a copy of patient data sitting on infrastructure you do not control, which is exactly the thing the whole exercise is meant to prevent. That is why BAA coverage for AI APIs is normally conditioned on running against zero-retention eligible endpoints.

The consequence that catches people is that eligibility is per endpoint and per feature, not per account. A developer turns on a capability that needs server-side state to work. It's a one-line change, it makes the product better, and it quietly moves that workload outside the terms your legal team negotiated. Nothing errors. Nothing warns. The dashboard looks the same.

Which is why the retention setting belongs in code review rather than in a compliance folder. If your gateway is the only thing that can call a model, you can assert the endpoint and refuse anything else in one place, and a developer who needs an ineligible feature has to come and talk to somebody. That conversation is the control. The BAA is just what makes the conversation worth having.

Worth knowing: zero retention usually costs you something. Some features genuinely need storage to function, and giving them up is a real product constraint rather than a formality. Decide that consciously and early, because finding out late means rebuilding something that already shipped.

Can You Just De-identify the Data and Skip All This?

Legitimately, yes. Data de-identified under the HIPAA Privacy Rule is no longer protected health information, so the rule stops applying and so does the need for a BAA on that path. It is a real strategy and for some workloads it is the right one.

In practice it is much harder than the sentence makes it sound, and the difficulty is specific rather than general.

The Safe Harbor method requires removing 18 categories of identifier. Names, geographic subdivisions smaller than a state, all dates more precise than a year that relate to an individual, phone numbers, email addresses, record numbers, device identifiers, biometric identifiers, full-face images, and any other unique identifying number or characteristic. Strip all 18 correctly and you are outside the rule.

Now consider what healthcare AI features are usually pointed at. Free-text clinical notes. Discharge summaries. Physician dictation. Message threads between a patient and a care team. Narrative text, written by humans under time pressure, where identity is carried in ways a field-level scrubber never sees. A rare condition combined with an age. A date that pins an admission. A facility name. A caregiver named in passing. The note that says the patient is a nurse at the same hospital.

So de-identification is engineering work with a measurable failure rate, and it needs evaluating like any other classifier, with a held-out set and somebody whose job is to find what it missed. Teams that treat it as a regex pass before the model call aren't de-identifying. They're lowering the density of identifiers and hoping.

Our honest read: use redaction as defence in depth on top of a BAA, not instead of one. The two together are strong. Redaction alone, on clinical free text, is a bet on your own scrubber that you'll be defending in an audit.

Where Does PHI Actually Leak in an AI Feature?

Through your own observability stack, more often than through the model provider. This is the section to read if you read nothing else, because it's the one missing from every other page on this topic.

LLM tracing tools, prompt monitors and evaluation platforms capture full prompt and completion text by default. That default isn't carelessness on their part, it's the entire product. A trace without the prompt is useless for debugging. So the tool does exactly what it promises, and what it promises is a searchable copy of every prompt.

The prompt is where the patient data is.

Which produces a genuinely absurd end state, and we've walked into it more than once on review. The model provider is under a BAA. The cloud is under a BAA. The database is encrypted, the access logs are clean, and the architecture diagram is correct. And protected health information is sitting in a third-party trace viewer, indexed and searchable, with a login half the engineering team has and nothing signed behind it.

The same pattern repeats in places nobody lists:

Error trackers. Most attach request bodies to exceptions automatically. Your model gateway throws a timeout, and the prompt goes with it into a tool some other team procured three years ago.

Prompt caches. Caching by prompt hash is a normal optimisation. It also means storing the prompt, or something derived from it, somewhere with its own retention policy that nobody wrote down.

Evaluation datasets. The best eval data is real production traffic, which is why teams build eval sets out of it. That set is now a permanent copy of patient conversations living in a repository, a bucket, or a notebook on somebody's laptop.

Support and debugging. An engineer pastes a failing prompt into a ticket. A product manager exports conversations to review quality. Both are disclosures. Neither shows up in any diagram.

Analytics. Someone wants to know what users ask about, so prompt text gets forwarded to a product analytics tool. Entirely reasonable request, and it moves patient data into a system chosen for funnel reporting.

None of these are exotic. Every one is a normal engineering practice that's correct everywhere except here, which is exactly why they survive review. Nobody flags the error tracker, because the error tracker was there long before the AI feature existed.

What Does a Compliant AI Feature Look Like in the Stack?

One boundary, drawn on purpose, with every crossing accounted for. The architecture is unremarkable. The discipline is in enumerating all of the hops instead of the interesting ones.

1. A single model gateway. One service owns every provider call. No other code holds a provider key. It's worth doing for reasons that have nothing to do with compliance, and then it turns out to be the thing that makes compliance tractable, because now there's exactly one place to assert the endpoint, apply redaction, and control what gets logged.

2. The agreement and the endpoint pinned together. A BAA with the provider, and the gateway asserting the zero-retention eligible endpoint in code rather than in a runbook. If a call cannot be made against an eligible endpoint, it fails loudly instead of falling back quietly.

3. Redaction before the call where the use case allows it. Not as a replacement for the BAA. As the second layer, so a misconfiguration is survivable.

4. Observability that logs identifiers, not bodies. Request ID, model, token counts, latency, outcome. Enough to debug a production incident, not enough to reconstruct a consultation. Where you genuinely need prompt-level tracing, put the tracing vendor under a BAA too and treat it as what it is, a system holding patient data.

5. Audit logs that answer the question an auditor asks. Who invoked what, when, against which record. Kept where you keep the rest of your audit trail.

6. Human review on anything clinical. If the output informs a care decision, a person signs off. That is a workflow requirement rather than a code one, and it is usually the constraint that shapes the product most.

The reason this list looks boring is that it is boring. Nobody needs a novel architecture here. What teams need is somebody to sit down with the actual request path and list every system that touches it. That takes about an afternoon, and it almost never happens, because it's nobody's job and it isn't interesting until it's urgent.

Who Does Not Need Any of This?

More people than are currently worrying about it, and we'd rather say so than let you spend a quarter on architecture you don't need.

HIPAA attaches to individually identifiable health information, held or transmitted by a covered entity or one of its business associates. Two conditions. If either one fails, none of this applies to you.

You don't need any of this if your AI feature never touches PHI. A marketing assistant drafting website copy. A coding assistant working on your own repository. An internal tool that summarises your product documentation. A support bot answering questions about billing policy in general rather than about a named person's bill. Those are healthcare companies using AI. They aren't healthcare AI, and the distinction is the whole ballgame.

You don't need this if you're not a covered entity or a business associate. A consumer wellness app that collects data directly from users and shares it with no provider or health plan is frequently outside HIPAA altogether. That doesn't mean unregulated, and here's where it stops being good news. State privacy law, the FTC Health Breach Notification Rule, GDPR if you have European users, and app store health data policies may all still reach you. Being outside HIPAA is a scoping fact rather than a free pass.

You probably don't need to self-host. See the next section. It's the most expensive answer to a question most teams haven't finished asking.

And if you're pre-launch with no real patient data yet, build the gateway and skip the rest for now. The gateway is the painful thing to retrofit, because retrofitting it means hunting down every provider call scattered across a codebase. The BAA, the redaction and the audit logging can all be added later against a single chokepoint, in about a week. Get the shape right and defer the paperwork honestly.

One thing worth saying out loud, because it affects who you should trust. If somebody quotes you for a compliance rebuild before asking which of those four situations you're in, you've learned something useful about them. The question takes two minutes and it changes the answer completely.

Should You Self-Host an Open Model Instead?

Usually not, and it's worth understanding why the instinct pulls so hard before you follow it.

The appeal is real. Self-hosting takes the third party out of the data path entirely, which genuinely settles the sub-processor conversation and makes the security questionnaire much shorter. If a contract or a jurisdiction says the data must never leave your infrastructure, this is the answer and there's no arguing with it.

The cost is that you've picked up the entire security burden the provider was carrying. Model serving infrastructure your team patches. GPU hosts your team monitors. Access controls, key management, audit logging and incident response, all of it yours now, on a system whose failure modes your team is meeting for the first time. For most teams a BAA plus zero retention with a major provider is both cheaper and easier to defend in an audit than a self-run deployment maintained by people whose actual job is the product.

There is also a question hiding inside the requirement. "The data cannot leave our infrastructure" is usually several different requirements wearing one sentence, and most of them are satisfied by documented cloud defaults rather than by hardware. We wrote that one up separately, because it deserves the room: on-premise LLM deployment, and whether you actually need it.

Work out which requirement you actually have before anyone quotes you for GPUs. It is the single most expensive assumption in this whole area.

How Do You Answer the Security Questionnaire?

With a sub-processor list and a data flow diagram. That is what the questionnaire is asking for, even when the wording is vague enough that it does not look like it.

The list names every third party that can see request content. Model providers. Hosting. Observability. Error tracking. Analytics, if prompt text reaches it. For each one, three facts: whether a BAA is in place, what the retention setting is, and whether your data trains anything.

The diagram shows where PHI enters, every hop it makes, where it comes to rest, and what is redacted at which point. One page. It does not need to be beautiful.

Here's the thing about that questionnaire. It's agonising when you have to work the answers out while filling it in, and it's twenty minutes when the boundary was drawn deliberately in the first place. The difficulty is never really the questionnaire. It's that the questionnaire is the first time anyone has asked the question, and it turns up with a deal attached to it.

That same document is the fastest audit you can run on yourself. Fill it in honestly before a customer sends it. Every question you can't answer is a piece of work you haven't done yet, and finding that out on your own schedule beats finding it out on theirs.

Where This Leaves You

You can put PHI in an LLM. Healthcare organisations do it in production every day and it's a normal, defensible thing to build. The vendors have made the contractual half genuinely straightforward, which is more than was true two years ago.

The half that fails is the half that happens after the model call, in tooling that was already there, chosen by people solving a completely different problem and solving it correctly.

So the work is smaller and less glamorous than most teams expect when they start reading about this. One gateway. One agreement with the endpoint pinned. One honest inventory of every system that can see a prompt. Redaction where it helps, never as the only defence. Then a diagram you can hand a customer without rehearsing first.

If you've already shipped and this article has made you uneasy, that's a reasonable response and it's usually fixable in days rather than quarters. The most common finding we see on review is a trace client with body capture left switched on, and the most common fix is a config change plus a retention purge. Not a rebuild. Be suspicious of anyone who tells you otherwise before they've looked.

Geminate Solutions builds healthcare and AI products as a development partner. You own the code from the first commit and we own delivery. If you want the data path reviewed, book a free 30-min scoping call and bring whatever you have, even if it is a whiteboard photo.