AI Medical Scribe: Production Guide

AI scribes look deceptively simple.

Record conversation. Transcribe conversation. Ask an LLM to write a note.

That is enough for a demo.

A product has to survive 40-minute visits, bad microphones, accents, medical terminology, interruptions, patient stories told out of order, half-finished sentences, several note formats and physicians who absolutely do not want to spend ten minutes correcting the thing that was supposed to save them time.

The scribe has three separate jobs

I like to separate the architecture into three layers.

1. Capture what was said.

Audio recording, upload, transcription, speaker handling and resilience to connection failures.

2. Understand what belongs in the clinical note.

Identify relevant facts, preserve negation, distinguish history from current complaint, understand medication changes, structure the encounter.

3. Produce the clinic’s documentation format.

SOAP note, specialty template, discharge summary, structured EHR fields or another required format.

These stages can use different models and rules.

Trying to solve all three with one giant prompt makes errors harder to trace.

Audio capture is product engineering, not AI magic

Long consultations create boring technical problems.

Browsers and mobile devices interrupt recordings. Networks disappear. Files become huge. Users close tabs. Microphones switch. Uploads fail at 97%.

A robust web scribe often needs chunked recording and upload rather than waiting until the end and praying a 45-minute file survives.

Compression matters too. Sending raw WAV for a long visit is usually unnecessary and expensive.

The user should also know whether recording is active, paused, uploading or failed. A red dot that looks reassuring while nothing is being saved is not a scribe feature.

Transcription quality is domain-specific

General speech models are very good now.

Healthcare still gives them creative opportunities.

Think of:

• drug names;
• rare diagnoses;
• physician names;
• local abbreviations;
• multilingual conversations;
• code switching;
• accents;
• brands and generics;
• laboratory terminology;
• specialty slang.

A local terminology layer can materially improve results even when you use a strong general transcription model.

For languages with smaller training corpora, a curated medical lexicon and real examples become even more valuable.

The transcript is not the note

This distinction matters.

A consultation transcript might contain:

“I stopped the tablets — no, sorry, the blue ones, I still take those, I stopped the other one after the cardiologist changed it.”

A clinical note needs to resolve that into a factual medication history or leave uncertainty visible.

The note-generation layer should not merely shorten the transcript.

It should map information into the clinic’s documentation structure while preserving factual meaning.

That is a reasoning task.

And because reasoning can fail, the physician needs a practical review workflow.

Do not throw away the raw evidence too early

When the generated note looks wrong, the fastest debugging path is often:

note → supporting transcript segment → audio

If the system only stores the final note, you lose the ability to answer:

• Was the audio wrong?
• Was transcription wrong?
• Did the model omit the fact?
• Did the model change the meaning?
• Was the template rule wrong?

Whether you retain audio and transcript, and for how long, is a privacy and product decision. But the architecture should make that decision explicit.

Negation and attribution are nasty little details

Healthcare notes contain phrases where one missing word changes everything.

“No chest pain.”

“Mother had breast cancer.”

“Patient denies taking warfarin.”

“Previous physician suspected MS; later workup was negative.”

A scribe should preserve:

• who the statement refers to;
• whether it is positive or negative;
• whether it is current or historical;
• whether it is confirmed, suspected or ruled out.

This is one reason I distrust scribe demos that only show beautifully formatted output.

Formatting is easy to admire.

Fact preservation is the actual product.

Specialty templates matter

A psychiatrist, orthopedic surgeon, IVF physician and dermatologist do not document the same consultation in the same way.

Even inside one specialty, clinics develop their own templates and habits.

A useful scribe should support structured rules such as:

• required sections;
• preferred wording;
• fields that should never be inferred;
• what belongs in assessment vs history;
• how measurements are formatted;
• which codes or structured fields are prepared;
• what is deliberately left for the physician.

This is where user-provided examples can help. A clinic can give the system representative protocols or notes, and the system can use them as controlled context — without changing the global safety rules.

Physician review is part of the product

If the doctor has to read the transcript and rewrite the note anyway, the scribe failed.

The review experience should make corrections fast:

• show a clear draft;
• highlight uncertain items when possible;
• preserve section structure;
• allow direct editing;
• make it obvious what will be written to the EHR;
• require approval before finalization where appropriate.

The right target is not “zero physician clicks.”

It is less physician cognitive and clerical work without losing control of the clinical record.

EHR integration changes the value dramatically

A standalone scribe that produces text in another browser tab can still be useful.

A scribe that knows the patient, appointment and note type, then writes the approved draft into the right place in the EHR is much more useful.

This is where the project stops being a model demo and becomes healthcare software.

Integration may use APIs, FHIR, vendor-specific interfaces, database access or other bridges depending on the system. See How to Add AI Without Replacing Your EHR.

What should the model never invent?

I would be extremely conservative about facts that were not present in the source or approved context.

For example, if the physician never discussed smoking status, the scribe should not fill in “non-smoker” because it makes the note look complete.

If a dose is unclear, it should not choose the most plausible one.

If the assessment is ambiguous, it should not convert uncertainty into a definitive diagnosis.

Completeness is not worth fabricated medicine.

How to evaluate a scribe

Do not grade only whether the note “sounds medical.”

I would test:

• factual recall;
• omissions;
• hallucinated facts;
• negation errors;
• speaker/attribution errors;
• medication name and dose accuracy;
• chronology;
• specialty-template adherence;
• useful compression;
• handling of uncertainty;
• time saved after physician correction.

The last metric is commercially important.

A note can score beautifully on a benchmark and still annoy physicians enough that nobody uses it.

Build or buy?

For many clinics, buying an existing scribe is absolutely the right answer.

There are mature products, and the basic transcription-plus-note use case is increasingly commoditized.

Custom development starts making sense when the scribe must become part of something larger:

• unusual local language or terminology;
• custom documentation protocols;
• deep EHR integration;
• automated chart preparation before the visit;
• post-visit actions;
• payer-specific documentation checks;
• proprietary clinical workflows;
• on-premise or unusual infrastructure requirements;
• one unified assistant covering more than scribing.

If all you need is a good SOAP note from a normal English consultation, I would test existing products before paying somebody to rebuild one.

If the transcript is only the first step in a custom clinical workflow, then architecture matters much more.