We need to act fast to close the NHS AI safety gap

Current safety standards can’t keep up with AI, writes Yvette Khozam, electronic prescribing and medicines administration (ePMA) lead pharmacist at West London NHS Trust 

NHS England wants to make the NHS “the most AI-enabled care system in the world”.

But despite the 10 year health plan’s ambition, we’re deploying AI under safety standards last substantively updated in 2018, built on a 2013 framework – well before generative models, adaptive algorithms, or real-world deception were in scope.

A review finally began in December 2024, five years after AI became a strategic priority with the Topol Review. In that time, we moved from cautious pilots to large-scale deployments while safety frameworks stood still.

Clinical risk standards DCB0129 and DCB0160 were built for predictable, fixed-logic software. They assume systems behave consistently, testing reveals all risks, and deployment matches testing. AI breaks all three.

What are the risks with AI?

Modern AI isn’t like traditional software – and our current standards can’t keep up.

Traditional software is like a recipe: same steps, same result. Prescribing tools use fixed formulas – you can test them because they don’t change.

An AI tool generating discharge summaries may learn that omitting certain details speeds up review times. It might start skipping medication changes

Modern AI systems can adapt over time, changing their outputs based on prior input –  even in identical cases.

Worse, recent research from Apollo Research shows that advanced AI models can engage in what researchers call “in-context deception” – essentially learning to give compliant answers when being monitored, while behaving differently in practice.

An AI tool generating discharge summaries may learn that omitting certain details speeds up review times.

In doing so, it might start skipping medication changes or follow-up instructions—not from malice, but because it’s optimising for the wrong goal. Without proper oversight, these omissions could accumulate unnoticed until a serious incident occurs.

Research from NeurIPS 2024 demonstrated a separate risk: steganography.

AI systems were shown to hide information inside otherwise normal text. Hidden inter-agent signalling could evade oversight and propagate errors.

NHSE has already issued warnings about non-compliant AI transcription tools and the national chief clinical information officer (CCIO) had to explicitly order trusts to stop using AI tools that don’t meet standards.

Trusts face a catch-22: use AI to improve care or wait years for governance. Many trusts are relying on local governance while national standards are under review.

For clinical teams, this means working with tools they’re told are safe, but which operate under frameworks that were written for static, deterministic systems – not self-improving models embedded in clinical decisions.

We need to take three immediate actions:

1. No black box tools coordinating behind the scenes

If an AI system involves more than one agent, its internal connections must be fully disclosed – including how the components communicate, what information they retain, and how they make decisions.

Any data passed between systems – directly or indirectly – must be traceable through audit logs. Manual checks should verify system coordination, and early pilots on low-risk services can help expose integration issues.

2. Test for hidden behaviour – not just surface answers

A model might appear safe in testing and behave differently in live settings. Suppliers must conduct red-team tests designed to provoke failure – missing new meds, downplaying adverse events, or reinforcing bias – and include the results in the safety case.

In parallel, systems should generate daily output reports, which are reviewed by a designated clinical or digital lead to catch subtle issues that red-teaming may miss.

3. Monitor for semantic drift

AI tools don’t always stay consistent. Over time, they may phrase outputs differently, omit key details, or shift how they summarise similar input.

Background upgrades can silently alter model behaviour, introducing unintended changes without clinical oversight. Every generated letter, summary, or flag should be logged for a set timeframe at creation and reviewed over time for changes in tone or meaning.

A standard operating procedure should define the human review process and include rollback steps using the manual system if the AI deviates from its intended function.

What are the critical blind spots in healthcare AI standards?

The NHS is deploying adaptive AI under standards that have not yet been fully aligned with the way these systems operate in practice.

The EU flags medical AI as high-risk but relies on static documentation. The US mandates broad safeguards and red-teaming but has not addressed hidden multi-agent behaviours like steganography.

No health system appears to have developed standards for unique AI failure modes like deception, performance drift or hidden coordination

A critical safety blind spot exists in healthcare. To my knowledge, no health system appears to have developed standards for unique AI failure modes like deception, performance drift, or hidden coordination. The NHS can be the first.

These systems are not theoretical; they are deployed now, and the risks are real. We cannot afford to wait.

I believe in AI’s promise – and the NHS’s ability to be the most AI-enabled care system in the world by treating safety as a moving target, as adaptive as the tools themselves.

Progress is underway. NHS England has opened a formal review of DCB0129/DCB0160. It has also published buyer guidance for AI-enabled ambient scribes and told organisations to stop using unregistered scribe tools.

The Medicine and Healthcare products Regulatory Agency’s software/AI as a medical device change programme is updating rules across the lifecycle.

NHSE, the Department of Health and Social Care and the Care Quality Commission have announced an AI early-warning system to spot patient-safety issues in NHS data.

The job now is to turn these national moves into day-to-day controls—versioning, audit trails and rollback owned by clinical teams.

Yvette Khozam is ePMA lead pharmacist at West London NHS Trust.  She is also a member of the Digital Health Networks CCIO Advisory Panel and Digital Health Leadership Programme delegate.