RAG in Healthcare: The Useful Version

RAG is one of those technologies that looks almost offensively simple in a demo.

Take documents. Split them into chunks. Create embeddings. Put them in a vector database. Retrieve similar chunks. Send them to an LLM.

Done.

Then you try it on a real medical record.

Now the patient has 14 years of history, duplicated diagnoses, scanned PDFs, labs, medications, amended notes, several physicians, records in two languages and three mentions of the same operation with slightly different dates.

Welcome back to healthcare.

What RAG is actually good for

Retrieval-Augmented Generation is useful when the model needs information that should come from an external source rather than its general training.

In healthcare that can mean:

• the patient’s record;
• clinic protocols;
• payer rules;
• internal policies;
• medical literature;
• device manuals;
• product information;
• local clinical guidelines.

The retrieval step finds potentially relevant source material. The LLM then uses that material to answer, summarize, classify or prepare another output.

The important word is potentially.

Retrieval is another probabilistic component that needs to be evaluated.

Similarity is not the same as relevance

Suppose the question is:

“What changed in the patient’s blood pressure treatment after the cardiology visit in May?”

A pure vector search may find semantically similar chunks about hypertension from three years earlier.

But we already know useful constraints:

• this patient;
• cardiology;
• around May;
• medication or treatment changes;
• probably after a specific encounter.

Good retrieval combines semantic similarity with structure and metadata.

That can include patient ID, document type, encounter, provider, date, code, specialty and other filters.

In healthcare, metadata is not administrative decoration.

It is part of meaning.

Patient scope should not depend on embeddings

This sounds obvious, but it deserves to be explicit.

Do not rely on semantic similarity to keep one patient’s data away from another patient.

Patient scope belongs in deterministic access control and filtering.

Same for tenant scope, organization scope, permissions and other hard boundaries.

The vector search decides what is relevant inside the allowed corpus.

It should not decide what the user is allowed to see.

Provenance matters

A model answer that says:

“The patient had a normal CT in March.”

is much more useful if the physician can see exactly where that statement came from.

For serious healthcare retrieval, I want to preserve a path back to the source:

• document;
• encounter;
• date;
• author/provider where available;
• source system;
• exact relevant excerpt or structured element.

This is important for trust and debugging.

It also helps distinguish:

source fact: “CT report dated March 14 says no acute intracranial abnormality”

from

model synthesis: “The patient’s imaging history is reassuring.”

Those are not the same thing.

Structured data should stay structured

One of the strangest RAG anti-patterns is taking beautifully structured healthcare data, flattening all of it into prose, embedding it, and then asking the model to rediscover the structure.

If you have a medication table with drug, dose, status, start date and end date, keep those fields available.

If you have laboratory values with analyte, unit, reference range and timestamp, keep them structured.

If the application needs “latest HbA1c,” a database query or FHIR search may be a better retrieval mechanism than vector similarity.

RAG does not mean every piece of information becomes a vector.

Often the best architecture is hybrid:

• structured query for known facts;
• semantic retrieval for narrative content;
• rules for explicit logic;
• LLM for synthesis.

Time is part of clinical meaning

Healthcare data changes.

A medication was active, then stopped.

A diagnosis was suspected, then ruled out.

A pathology report was preliminary, then amended.

A patient had one insurance policy last year and another now.

Retrieval that ignores time can return a perfectly relevant but clinically obsolete fact.

So temporal logic matters:

• effective date;
• authored date;
• encounter date;
• status;
• version/amendment;
• “current” versus historical.

This is one reason generic document-chat systems often feel impressive for five minutes and unreliable after a week of real use.

Chunking medical records is not a clerical detail

A chunk should preserve enough context to remain meaningful.

Splitting a note in the middle of a medication plan can destroy the relation between the drug and the instruction.

Mixing several unrelated sections into a giant chunk reduces retrieval precision.

Useful strategies depend on the source:

• section-aware chunking for clinical notes;
• document-level metadata for reports;
• structured indexing for labs and medications;
• encounter-aware grouping for longitudinal history;
• separate embeddings for different semantic representations where the use case justifies it.

The correct unit of retrieval is a product decision, not a universal token count.

More context can make the answer worse

LLM context windows have become very large.

That does not mean the best retrieval strategy is “send everything.”

Too much context increases cost and latency, but the bigger issue is attention.

If you send 70 pages of medical history for a question that needs three facts, you are asking the model to find the needle again after your retrieval system already had the opportunity to do it.

A good RAG system should make the model’s job easier.

Not proudly demonstrate that the context window is enormous.

The knowledge base also needs lifecycle management

Clinic protocols change. Payer rules change. Staff uploads a new PDF but forgets to remove the old one. A policy is superseded.

A production RAG system needs concepts such as:

• version;
• effective date;
• source owner;
• approval status;
• expiration/supersession;
• re-indexing;
• deletion;
• monitoring for stale or failed ingestion.

Otherwise your “knowledge base” becomes a very sophisticated folder called final_v7_really_final.pdf.

RAG reduces hallucination. It does not abolish it.

Grounding the model in relevant sources is one of the best ways to improve factual reliability.

But the model can still:

• misread the retrieved text;
• combine two facts incorrectly;
• overstate what the source says;
• answer beyond the evidence;
• ignore a more relevant source;
• retrieve the wrong thing in the first place.

So important workflows should make uncertainty and source provenance visible.

For clinical or other high-consequence outputs, human review may remain appropriate even with excellent retrieval.

How I test healthcare RAG

I do not start with “Does the answer look good?”

I separate the system into stages.

Retrieval test: Did we retrieve the right source material?

Coverage test: Did we miss a fact required to answer?

Contamination test: Did we retrieve irrelevant or wrong-patient content?

Generation test: Given the correct evidence, did the model use it accurately?

Citation test: Can the user trace important claims back to the source?

Temporal test: Did the system distinguish current from historical information?

Permission test: Could a user retrieve information outside their allowed scope?

That makes debugging much easier than arguing with the final paragraph.

When RAG is the wrong tool

Do not use RAG because your architecture diagram needs a vector database.

If the task is:

“What is the patient’s next appointment?”

query the appointment system.

If the task is:

“Is the current balance above $500?”

query the billing data.

If the task is:

“Does this claim contain all seven required fields?”

use explicit validation where possible.

RAG becomes valuable when meaning is trapped in language or when users need to find relevant information across a large corpus.

It is not a replacement for databases, APIs or business logic.

A healthcare RAG architecture I actually like

The pattern is usually some variation of:

source systems → normalization/metadata → permission filter → structured + semantic retrieval → reranking → LLM → source-linked output

Depending on the project, the sources may be an EHR, FHIR server, document store, payer library or internal knowledge base.

The important design principle is that the path to the original fact remains visible.

Healthcare AI gets much more useful when we stop treating data as a pile of text and start treating it as a connected record of events, entities and sources.

That is the direction I have been pushing healthcare architecture for years.