AI Agents in Healthcare: Production Guide

A chatbot answers a question.

An agent can do something about it.

That difference sounds small until the thing it is doing is changing a real appointment, opening a patient record, sending a preparation protocol or writing information back into an EHR.

Then suddenly “the AI answered correctly” is not a sufficient test.

The system also has to understand which patient, which clinic rule, which tool, which data source, which action is allowed, whether the action actually succeeded, and when it should stop and ask a human.

That is the interesting part of healthcare agents.

Not the chat bubble.

What is an AI agent in healthcare?

I use a fairly practical definition:

A healthcare AI agent is software that can interpret a request, retrieve relevant context, choose from a bounded set of actions, execute approved tools and continue the workflow based on the result.

For example, a patient writes:

“I need Dr Cohen next week, preferably after work.”

A production booking agent may need to:

1. resolve who the patient is;
2. understand what service they are asking for;
3. check whether Dr Cohen provides that service;
4. translate “after work” into a useful search window;
5. query actual availability;
6. offer two or three slots rather than dumping a calendar;
7. handle “anything earlier?” without forgetting the previous request;
8. confirm the selected time;
9. call the booking tool;
10. verify that the booking succeeded;
11. send the clinic’s approved confirmation and preparation information.

That is an agent workflow.

The LLM handles interpretation and reasoning. The clinic systems still remain the source of truth for facts and transactions.

The model is not the architecture

This is the mistake I see most often in early AI products.

The architecture diagram is basically:

Patient → GPT → EHR

There are several missing boxes in that diagram.

A real implementation normally needs some combination of:

• identity and session handling;
• clinic context and configuration;
• patient-specific context;
• retrieval from medical or administrative sources;
• tools with narrow permissions;
• deterministic business rules;
• state across several turns;
• tool-result validation;
• audit logs;
• escalation paths;
• monitoring and regression tests.

If all of that logic is hidden inside one giant system prompt, you do not have a simple architecture.

You have a complicated architecture stored in the least convenient place to debug it.

Context is more than the current message

Patients speak as if they are speaking to a person.

Which means they say things like:

“Yes, that one.”

“Actually Thursday is better.”

“What about the other doctor?”

“Do I need the referral for it?”

Those messages make perfect sense if you remember the previous conversation.

They are almost meaningless if every turn starts from zero.

A useful agent needs several types of context, and they should not all be treated the same way:

Clinic context — opening hours, policies, addresses, communication rules.

Provider context — specialties, services, schedules, languages, restrictions.

Service context — duration, price, preparation, allowed providers, booking rules.

Patient context — identity, relevant appointments, payer, conversation history and other data the workflow is allowed to use.

Conversation state — what the patient is currently trying to achieve and what has already been established in the current flow.

One important design principle is that retrieved context should normally inform reasoning, not silently expand authority. Knowing something and being allowed to act on it are two different questions.

Tools should be boring

I like the reasoning layer to be smart.

I like the transaction layer to be boring.

A booking tool should do something very specific such as:

book_appointment(patient_id, provider_id, service_id, slot_id)

It should validate its inputs, check permissions, perform the transaction and return an explicit result.

The model should not be writing arbitrary SQL because it “understands the database.”

It should not decide that a cancellation API probably behaves like the booking API.

And it should not infer success because the tool response looked optimistic.

This separation is one of the easiest ways to make an agent safer and easier to debug.

The agent should try — but not at all costs

A badly designed agent has two common personalities.

The first gives up too quickly:

“No appointments found. Please contact the clinic.”

The second behaves like an overenthusiastic salesperson with root access.

A better agent has a search strategy.

If the requested doctor is unavailable, can it offer another provider who performs the same service?

If there is nothing at 18:00, can it ask whether afternoon is acceptable?

If the patient says “earlier,” can it search earlier slots without starting the whole conversation again?

If a service is expensive, can it explain an approved pricing fact from clinic context rather than inventing a discount?

The goal is useful persistence within boundaries.

That is very different from “complete the task no matter what.”

The most important skill is knowing when not to answer

Healthcare agents regularly encounter questions that look administrative but are actually medical.

For example:

“Is this referral enough for the procedure?”

“Can I take the medication before the test?”

“Is this doctor suitable for my condition?”

The clinic may have enough context to answer some of these administratively. Others need medical staff.

The agent should have an explicit policy for that distinction.

A generic fallback such as “I don’t know” is not enough. A good escalation includes the context the human needs and tells the patient what happens next.

“I’m passing this question to the medical team. They will get back to you.”

Simple sentence. Important architecture.

Patient-facing agents need identity boundaries

If the agent only answers public questions such as opening hours, identity may not matter.

The moment it starts discussing a patient’s existing appointments, documents, balances, medical information or other private data, it does.

The important rule is not “identify every user immediately.”

It is:

Do not cross from public/general information into patient-specific information or actions without the level of identity assurance appropriate for that workflow.

This is one reason patient communication agents need workflow design, not just prompts.

Write access changes the risk class of the project

There is a huge difference between:

• reading a clinic FAQ;
• summarizing a document;
• suggesting a possible appointment;
• actually booking it;
• modifying the medical record;
• making a clinical decision.

The farther down that list you go, the more I want deterministic validation, explicit permissions, auditability and human control.

A system that can only draft a note can still hallucinate, but a physician can review it before anything becomes part of the record.

A system that autonomously changes orders, medication or clinically meaningful information needs a completely different risk discussion.

Agents fail in very ordinary ways

The exotic AI failures get attention.

In production, some of the nastiest failures are boring:

• a tool returns an empty result and the model interprets it as “none exist”;
• the patient changes intent and the old booking state survives;
• a provider name is transliterated differently;
• availability changed between search and confirmation;
• the model calls the correct tool with the wrong identifier;
• a previous patient’s context leaks into a new session;
• a tool times out after completing the action, so the agent tries again;
• the clinic changed a price but the old context is still being retrieved;
• a fallback message hides the fact that the workflow is stuck.

This is why healthcare AI agent evaluation deserves its own discipline.

Where healthcare agents are genuinely useful

I would look first at workflows with three characteristics:

1. a lot of language — somebody has to interpret free text, calls or documents;
2. a bounded operational goal — schedule, collect, route, summarize, prepare, check, remind;
3. systems the agent can safely read or act on.

Good examples include:

• patient communication and appointment management;
• referral and intake processing;
• document classification and routing;
• pre-visit preparation;
• chart preparation for physicians;
• post-visit follow-up;
• call and conversation quality analysis;
• administrative eligibility or document completeness checks;
• internal staff assistants that can find and prepare information across systems.

The more a workflow depends on open-ended clinical judgment, the less I want an autonomous agent pretending the boundary does not exist.

How I would design the first version

Do not begin by asking:

“How autonomous can we make it?”

Begin with:

“What is the smallest useful set of actions we are willing to let it perform?”

Then define:

• the sources of truth;
• the allowed tools;
• what requires confirmation;
• what requires identity;
• what always escalates;
• how state is stored;
• what gets logged;
• what the human sees when the agent escalates;
• how you will test it before launch.

Then make the agent good inside that box.

You can always widen the box later.

It is considerably less fun to discover in production that the box never existed.