We can fix the hidden flaws in ambient voice technology

We need to get the governance right for AI scribing, writes Yvette Khozam, chief pharmacy information officer at Mid and South Essex NHS Foundation Trust and Essex Partnership University NHS FT

NHS England has re-opened its self-certified registry for (AVT), a signal that AI scribing is heading for scale, aligned to the 10 year health plan’s ambition for AI to become every nurse’s and doctor’s ‘trusted assistant’.

We’re scaling under assurance models based on data security standards (DSPT) set in 2016, before AI tools started reading from the record and drafting back into clinical workflows.

The shift toward a stronger national cyber framework (CAF) since September 2024 is welcome and version 4 has begun including AI-related risks.

But I keep returning to a question the current framework doesn’t yet seem to answer. What happens when the record itself becomes part of the prompt?

More than transcription

DSPT asks: did we keep the data safe? CAF asks: did the system withstand attack?

Neither yet seems to ask: did the record steer the output?

AVT is a useful lens, and one I know first-hand, having written everything from evaluation criteria to assessing integration requirements.

It used to be transcription. Now it’s ‘listen + read + draft’.

Transcription records what you say. Context-aware tools predict what comes next. And like autocomplete, they can be confidently wrong.

We’re no longer securing a static recording of what was said. We’re securing a tool that generates new text

As the National Cyber Security Centre has warned, current large language models “do not enforce a security boundary between instructions and data inside a prompt”. In a clinical record, that’s a direct patient safety risk.

We’re no longer securing a static recording of what was said. We’re securing a tool that generates new text from what it reads.

Indirect prompt injection

A referral letter arrives by email. The trust system doesn’t talk to the external record, so it gets copy-pasted in. But text can carry hidden instructions that are invisible to people but not to AI.

If an AI tool reads that text, it may follow those instructions, whether planted deliberately or arrived through messy formatting.

The result? A clinical note that’s subtly wrong but not wrong enough for anyone to notice or patient data sent somewhere it shouldn’t go.

We need to close the gap. If systems talk to each other, clinicians don’t need to copy and paste. Every integration that replaces a manual workaround shrinks the risk.

Sensitive information disclosure

A clinician opens an AI scribe to pull together a patient summary before a consultation. The summary comes back with information the clinician wouldn’t normally be able to see.

This might happen if vendors give the tool more access than the user has, setting it up as ‘clinician’ rather than matching what that individual user is allowed to view.

That’s exactly the kind of access an attacker would look for. And if a prompt injection hits a tool with that reach, the damage isn’t limited to one bad output. It’s a way into the wider record.

One wrong note gets saved as fact, and every system and clinician downstream builds on it

The tool’s access should mirror the individual clinician’s permissions, not a blanket role. Vendors should be required to demonstrate this at procurement.

If AI-drafted notes are added to the system without a label saying this was AI-generated, it becomes future input. One wrong note gets saved as fact, and every system and clinician downstream builds on it.

The fix is clear: tag every AI output with what generated it, when, and from what. If a flaw emerges, you will then be able to find every record it touched.

Without provenance, you’re trawling blind. With it, you have an audit trail.

Foundations for progress

The EU AI Act and the NIST AI Risk Management Framework in the US represent significant progress on AI governance, but neither thoroughly addresses how these risks play out at the clinical frontline.

The US has begun developing dedicated AI cybersecurity guidance for the health sector, but even that is still in preview. To my knowledge, no health system has yet published assurance standards that treat these as routine clinical cyber risks.

CAF alignment is underway, the Information Commissioner’s Office has consulted on generative AI, the government has published an AI cyber security code of practice, and NHS England has begun direct supplier engagement on cyber compliance.

NHS England’s AVT supplier registry could, with the right conditions, become the governance lever this needs. If we get this right for ambient scribing, it will become the template for every generative AI tool that reads from and writes to the patient record.

The 10 year health plan wants AI to become every clinician’s trusted assistant, but trust is earned.

We need cleaner inputs, clear provenance, and least-privilege access. That’s how we will earn trust.

Khozam will be speaking in the panel session ‘Creating a culture of digital clinical safety’ on the Digital Frontline Stage at Digital Health Rewired 2026.

Rewired takes place at the NEC Birmingham on 24 – 25 March 2026. Register here.