LLMs, PHI & HIPAA: Architecture Guide
By Alex Mastryukov · Last updated: October 4, 2026
A question I hear regularly is:
“Can we send patient information to an LLM and still be HIPAA compliant?”
The useful answer is not “AI is HIPAA compliant” or “AI is not HIPAA compliant.”
Software products are not blessed by a regulatory priest and permanently declared compliant.
You have to look at what data is being processed, by whom, under what agreement, for what purpose, with what safeguards and inside which workflow.
This guide is about architecture and implementation, not legal advice. For the final compliance position, involve counsel and your privacy/security team.
Start with the boring question: is this PHI?
Before discussing models, identify the data.
If information is protected health information under HIPAA and your organization is a covered entity or business associate, the normal HIPAA requirements follow the data into the AI workflow.
That includes prompts, retrieved context, uploaded documents, transcripts, model outputs, logs and caches if they contain PHI.
One recurring mistake is protecting the source EHR beautifully and then writing the complete patient prompt into an application log visible to half the engineering team.
The AI call is only one hop in the data path.
A cloud provider processing ePHI may be a business associate
HHS guidance is very clear about cloud services: a cloud service provider that creates, receives, maintains or transmits ePHI on behalf of a covered entity or business associate is generally a business associate, including in some cases where the provider only stores encrypted ePHI.
That normally means a HIPAA-compliant Business Associate Agreement is required and the parties need to meet their respective obligations.
See the official HHS Guidance on HIPAA & Cloud Computing and HHS Business Associate guidance.
This is why I do not like the phrase:
“The model is HIPAA compliant.”
The real question is whether the specific service, account, contract and technical configuration you are using fit the required arrangement.
A BAA is necessary in many architectures. It is not sufficient.
Signing a BAA does not automatically make a bad system good.
You still need the normal security work:
- risk analysis;
- access controls;
- authentication;
- appropriate encryption;
- logging and monitoring;
- incident response;
- backups and recovery where relevant;
- retention and deletion policy;
- workforce access rules;
- subcontractor review;
- minimum-necessary thinking;
- technical and organizational safeguards around the complete workflow.
A BAA is a contract.
It is not an architecture diagram.
Map the whole PHI path
For an LLM workflow, I want to draw the actual path of the data.
For example:
EHR → integration service → retrieval layer → application → model API → application → audit log → EHR
Then ask at every point:
- Does PHI exist here?
- Is it stored or only transmitted?
- For how long?
- Who can access it?
- Is it copied into logs?
- Is it used for model training or service improvement under the applicable terms?
- What subprocessors are involved?
- How is it deleted?
- What happens when the customer relationship ends?
This exercise usually finds more risk than staring at the model provider’s homepage.
Do not accidentally put PHI into observability tools
Modern software loves telemetry.
Developers log requests. Error tools capture payloads. Session-replay products record screens. Analytics systems capture form fields. Support tools collect screenshots. Prompt-observability platforms store complete conversations.
All of those may be useful.
They may also become additional PHI processors.
One of the first things we review in a healthcare AI architecture is where prompts, retrieved context and outputs are being logged. Development convenience can quietly create a second copy of the medical record in a tool nobody originally considered part of the healthcare system.
RAG does not remove the HIPAA problem
Sometimes people assume Retrieval-Augmented Generation is safer because the model is not permanently trained on the patient’s data.
That is not the right distinction.
If PHI is retrieved and sent to the model as context, PHI is still being processed.
RAG can improve factual grounding and data minimization because you can retrieve only the information needed for the task. That is useful.
But “we use RAG” is not a compliance exemption.
Read more about the architecture in RAG in Healthcare.
Data minimization is an architectural feature
A physician preparing for a cardiology visit may not need the patient’s entire 12-year record sent to the model.
A scheduling agent normally does not need clinical notes at all.
A documentation-audit workflow may need a very specific subset of records.
Good architecture narrows context to what the workflow needs.
That reduces privacy exposure, lowers token cost and often improves model quality because irrelevant context is not drowning the useful facts.
This is one of those rare cases where security, cost and AI performance all want approximately the same thing.
Separate patient-facing administrative context from clinical context
I am particularly cautious about giving patient communication agents broad medical-record access “just in case.”
If the agent handles scheduling, pricing and preparation instructions, its default context should be administrative.
If a specific workflow genuinely needs clinical data, retrieve that data deliberately and only after the appropriate identity and authorization checks.
Do not give an AI receptionist the entire chart because someday a patient might ask a medical question.
That is not flexibility.
That is unnecessary blast radius.
Model outputs can contain PHI too
The output is not harmless because the model created it.
If an LLM produces a summary containing the patient’s diagnoses, medications and identifiers, that output is still sensitive information inside the workflow.
The same access, storage and disclosure thinking applies downstream.
For example, a generated note copied into a ticketing system may create a new disclosure even if the model call itself was perfectly configured.
De-identification can change the problem — if it is real de-identification
For some analytics, research or model-evaluation workflows, you may not need identifiable information.
HHS distinguishes properly de-identified information from PHI. If data has actually been de-identified under the applicable standard, the HIPAA treatment is different.
But replacing the patient’s name with “Patient 123” while leaving dates, rare diagnoses, addresses and other identifying data in the text is not automatically de-identification.
Do not use the word as decoration.
What about employees using public ChatGPT with patient data?
This is a governance problem many healthcare organizations already have whether they officially bought AI or not.
People discover useful tools faster than policy committees write policies.
The answer is not pretending nobody uses them.
The organization needs to understand:
- which AI tools staff are allowed to use;
- which accounts or plans are approved;
- what data may be entered;
- what contractual protections exist;
- how staff should handle files and screenshots;
- what to do when an approved workflow does not cover the real work people need to perform.
A prohibition that ignores the operational reason employees started using AI often creates shadow AI rather than eliminating it.
A practical reference architecture
For many healthcare AI projects, a sensible pattern looks like this:
- Keep the EHR or operational system as source of truth.
- Retrieve only the context required for the task.
- Use an approved model endpoint under the required contractual arrangement.
- Keep application-level authorization outside the model.
- Restrict tools to explicit actions and scopes.
- Log enough to audit the workflow without spraying PHI into every telemetry product.
- Store outputs only where the business workflow actually needs them.
- Define retention and deletion across all processors.
- Test failure paths as aggressively as the happy path.
The exact implementation depends on the organization and jurisdiction, but those principles travel well.
HIPAA is not the only question
A US healthcare project may also involve state privacy law, medical-device rules, consumer-health-data rules, contractual obligations, security standards and sector-specific requirements.
Outside the US, HIPAA may not apply at all while GDPR, the EU AI Act, Israeli privacy law or other frameworks do.
Our broader overview is on the Healthcare AI Security & Compliance page.
The rule I use
Do not ask:
“Can AI process PHI?”
Ask:
“Can this specific architecture process this specific data for this specific purpose under the agreements, safeguards and controls we actually have?”
That question is less exciting.
It is also the one that survives contact with a compliance review.