Testing Healthcare AI Agents

A beautiful AI demo proves one thing:

Somebody found a conversation where the system looked beautiful.

That is useful.

It is not validation.

Production agents need repeatable evaluation, because the same flexibility that makes language models useful also makes them capable of failing in wonderfully creative ways.

Healthcare adds one more complication: the failure may not be obvious from the final sentence. The message can sound perfect while the agent called the wrong tool underneath it.

Start with scenarios, not vibes

For an appointment agent, a scenario might define:

Setup: patient has no appointment; Dr Cohen has two slots Thursday; service requires 30 minutes.

Patient: “Can I see Dr Cohen Thursday afternoon?”

Expected behavior: offer the two actual slots.

Required tool behavior: call availability search with Dr Cohen + correct service + Thursday afternoon.

Forbidden behavior: do not claim a slot before tool results; do not book without confirmation.

Then add the next turn:

Patient: “Anything earlier?”

Now we are testing whether the agent preserves the service/provider context while changing the time constraint.

That is much more useful than asking a tester whether the conversation “felt smart.”

Evaluate several layers separately

I like to split failures into categories.

1. Intent and language understanding

Did the agent understand what the user wanted?

Examples:

• booking vs cancelling;
• “next Tuesday”;
• “after work”;
• “the other doctor”;
• “same as last time”;
• a misspelled provider name;
• transliterated service names;
• mixed-language input.

2. Context and state

Did it remember what should persist?

Did it forget what should have been reset?

This is where you test:

• previous turn state;
• patient identity;
• previous conversations where allowed;
• changed intent;
• changed provider;
• changed service;
• new session isolation.

3. Retrieval

Did it retrieve the correct clinic, provider, service, payer or patient information?

A fluent answer based on the wrong context is still wrong.

4. Tool choice

Did it call the correct tool?

Did it call a tool when it should have answered from trusted context?

Did it answer from memory when it should have checked the source system?

5. Tool arguments

This is a separate failure class.

The agent may correctly decide to search slots but send:

• wrong patient;
• wrong provider;
• wrong service;
• wrong date range;
• wrong clinic location.

The tool choice was correct. The action was not.

6. Transaction result

Did the operation actually succeed?

A model saying “You’re booked” does not make an appointment exist.

Evaluation should inspect system state when possible.

7. Safety and escalation

Did the agent identify a question it was not allowed to answer?

Did it escalate rather than hallucinate?

Did it expose private information before identity?

Did it avoid taking a consequential action without required confirmation?

8. Language quality

Healthcare agents are customer-facing software.

Test:

• grammar;
• gender handling where the language requires it;
• unwanted mixed language;
• strange Unicode or foreign words;
• formatting on WhatsApp/SMS;
• professional but not robotic tone;
• whether the answer is unnecessarily long.

Build fixtures that represent the real world

An evaluation suite is only as good as its fixtures.

If every scenario contains a patient called John Smith, one provider and three clean appointments, the agent will become very good at your imaginary clinic.

Useful fixtures include:

• several providers with overlapping services;
• unavailable requested provider;
• same provider at multiple locations;
• payer differences;
• missing price;
• stale or conflicting context;
• existing appointments;
• similar names;
• no matching service;
• incomplete patient identity;
• tool errors;
• empty search results;
• changed availability between search and booking.

I want the fixture to create the kind of ambiguity the production system will actually meet.

Test conversations, not only single turns

Single-turn benchmarks miss some of the most important agent failures.

Consider:

Patient: “I need a dermatologist next week.”

Agent: offers Monday and Wednesday.

Patient: “Wednesday.”

Agent: asks which time.

Patient: “Actually, is there anything with Dr Levi?”

What should persist?

The requested specialty probably persists.

The selected Wednesday slot should not.

The provider constraint changes.

That kind of state transition is exactly where agents become fragile.

Multi-turn evaluation is mandatory for multi-turn products.

Include adversarial and weird cases

Real patients are more creative than QA teams.

Test things such as:

• “Book whatever is earliest and don’t ask me anything”;
• contradictory instructions;
• a pasted medical report inside a booking conversation;
• voice-message transcription with errors;
• emoji-only response;
• patient asking the agent to ignore clinic rules;
• prompt-injection text inside an uploaded document;
• user claiming to be clinic staff;
• repeated message after a timeout;
• request to reveal another patient’s information.

You are not trying to make the model fail for entertainment.

You are discovering which failures need deterministic controls.

Tool failures deserve their own suite

Happy-path tools are easy.

Test:

• timeout before execution;
• timeout after execution;
• 401/403 authentication failure;
• 500 response;
• malformed result;
• duplicate request;
• partial success;
• source system unavailable;
• rate limit;
• success response with stale state.

The agent should react differently depending on what happened.

“Try again” can be dangerous if the first booking succeeded but the response timed out.

This is where idempotency and transaction design matter more than prose quality.

Define invariants

Some properties should not be graded subjectively.

Examples:

• never return patient-specific data before required identity;
• never call book_appointment without a valid slot ID;
• never book before confirmation;
• never answer defined medical categories without escalation;
• never cross tenant boundaries;
• never invent availability;
• never claim a transaction succeeded unless the tool confirms success.

These are invariants.

I like them because they turn safety requirements into tests.

Use both deterministic checks and LLM grading

Some things are easy to check programmatically:

• correct tool called;
• expected database state;
• forbidden tool not called;
• output contains only allowed characters;
• response below a maximum length;
• escalation record created.

Other things need semantic judgment:

• was the response helpful?
• did it answer the question without overexplaining?
• did it preserve meaning in another language?
• did it offer reasonable alternatives?

LLM-as-judge can be useful for the second category, but I would not use a judge model as the only test of transactional correctness.

Run regressions after every meaningful change

Agent development has a nasty property:

You fix scenario 47 and accidentally break scenario 12.

Prompt changes, model changes, tool-description changes and context changes can all alter behavior somewhere else.

That is why the suite becomes more valuable over time.

Every production bug should ask a question:

“Can we turn this incident into a regression test?”

If yes, the system gets harder to break in the same way twice.

Measure more than pass/fail

Useful metrics can include:

• scenario pass rate;
• tool-call accuracy;
• unsafe-action rate;
• escalation precision and recall;
• booking completion rate;
• hallucinated-fact rate;
• average turns to completion;
• recovery rate after tool failure;
• language-quality defects;
• context-leak incidents;
• latency and cost.

Do not optimize one metric blindly.

An agent can achieve a wonderful safety score by escalating every single question to a human.

Congratulations. You built email forwarding.

Production monitoring completes the loop

A test suite will never contain everything real users do.

So production needs monitoring too.

Useful practices include:

• sample and review conversations;
• automatically flag unusual or failed flows;
• track escalation reasons;
• identify new intents;
• collect tool failures;
• compare completion rates by version/model;
• add new production failures back into the eval suite.

That creates a cycle:

production → incident/example → new test → fix → regression suite → production

This is how the agent becomes more reliable instead of merely accumulating prompt text.

The standard I use before launch

I do not need an agent to be perfect.

No human receptionist is perfect either.

I need to know:

• what it is good at;
• where it fails;
• which failures are contained;
• which actions are protected by deterministic checks;
• how quickly problems become visible;
• whether the human handoff works.

That is a much more honest production-readiness standard than:

“We chatted with it for an hour and it seemed really smart.”

NIST’s AI Resource Center uses the broader term TEVV — testing, evaluation, verification and validation — for this kind of lifecycle thinking. Healthcare agents need the same discipline, translated into the very concrete workflows they are allowed to perform.