Clinicians wouldn’t ‘just try’ an experimental drug — so why AI?

Clinical decision support needs the same robust governance as medicine, writes Jordan Fulcher, clinical solutions consultant – EMEA at Wolters Kluwer

We don’t tolerate unapproved medicines or even technology solutions that are outside of the framework, so why are we not doing more to support clinicians with validated AI for clinical decision support?

One formative experience that still sticks with me is back from when I was a newly qualified pharmacist working my first overnight on call.

It’s 1am, the bleep goes off, and it’s the intensive care unit. The registrar had a question about treatment options and dosing that I immediately knew was going to take some time to research, but it was urgent so checking the next day wasn’t an option.

They didn’t want to wake their consultant. So, they called the very junior, on-call pharmacist.

Shadow AI is what happens when people use unapproved, unsupervised AI tools for real clinical work

Ten years ago, I had to rely on the resources provided in the on-call folder and the world wide web. Open the guidelines, dig out the primary literature, disappear into PubMed, cross-check dosing, interactions, renal function, the lot and then call back with something that I hoped made sense.

In fact, we both agreed to go off separately, do some research and call back in 30 minutes. Luckily, we reached a consensus we were both happy with.

But in 2026, could I honestly say I wouldn’t reach for an AI-powered deep research tool if I found myself in the same situation? When you’re under pressure, stressed, overworked, you try to work as efficiently as possible.

This would be a prime example of shadow AI. It’s what happens when people use unapproved, unsupervised AI tools for real clinical work, most often for decision support, because it’s fast, familiar and sitting on their phone.

Shadow AI

The existing IT and clinical systems have checks and balances: approved references, local policies, clinical governance, information governance, clinical safety sign-off. Shadow AI bypasses all of that.

It sits outside procurement, outside safety cases, outside data protection impact assessments, outside audit trails.

The problem is often compounded by the fact that the greatest need for these kinds of tools, often occurs in some of the higher-pressure situations. As clinicians we want an answer, and with the “answer engines” on our phone, we get one.

One of the growing concerns amongst teachers and supervisors, beyond the risks of hallucination and error, is that they’re not equipping clinicians to practice.

Clinical decision-making, especially in acute settings, is rarely a single question with a single answer

They take the question at face value, search broadly, and deliver a plausible answer. But clinical decision-making, especially in acute settings, is rarely a single question with a single answer.

Clinicians should learn to work iteratively. To always ask “what else?” or “what next?” before narrowing down on the final answer. This is why many large language models (LLMs) struggle to triage effectively.

A recent study in Nature showed that ChatGPT under triaged in 52% of emergency situations.

If you asked a senior clinician, “Is this a side effect from medication X?”, an expert doesn’t simply list known adverse effects.

They’d ask clarifying questions such as: “When did the symptom start relative to initiation or dose change? What’s the dose and route? Renal and hepatic function? Any interacting medicines? What else changed today? What’s the baseline? What are the red flags that would make you treat this as a new differential diagnosis?”

An answer engine doesn’t reliably do that mentoring work, and under pressure it can falsely push the clinician to a simplistic answer.

We created governance because we recognised that under pressure, humans will reach for the fastest route to certainty. AI is no different

In medicine, we already recognise that public LLMs shouldn’t be used routinely or independently to answer medicines-related questions, and any clinical copilot needs to be able to replicate that iterative thought process.

Governance issues

This is a system problem, at a national and hospital level.

We created governance because we recognised that under pressure, humans will reach for the fastest route to certainty. AI is no different.

The NHS has already shown it can respond sensibly when a technology moves quickly such as with ambient scribing. National guidance sets out organisational responsibilities and makes clear that ad‑hoc, unsupervised use is no longer acceptable.

That’s the direction we need for decision support too. Shadow AI is happening because it meets a real need, not because clinicians are reckless.

We need an approved, supported route for decision support that’s aligned with the existing path of least resistance

The job now is to define “safe”, provide approved tools and workflows, and give staff a supported way to use generative AI without gambling with patient trust or data.

This won’t be solved by telling clinicians what not to do at 1am.

We need an approved, supported route for decision support that’s aligned with the existing path of least resistance. On a mobile, in an electronic patient record, and accessible.

That’s the kind of maturity we’ve begun to establish elsewhere, and it needs to extend to decision support too.

In pharmacy, we don’t rely on good intentions alone, we rely on formularies and pharmacovigilance. AI should have to earn that same trust.