Doctors raise safety concerns about AI discharge summary tool

Doctors groups have raised safety concerns over an AI tool that writes hospital discharge letters, which is planned to be rolled out nationally through the federated data platform (FDP).

The technology, developed at Chelsea and Westminster Hospitals NHS Foundation Trust, uses a large language model to extract key details from medical records, such as diagnoses and test results, to help write the documents needed to discharge people from hospital.

The AI-Assisted Discharge Summary tool has been in use at the trust since August 2025.

However, Professor Victoria Tzortziou Brown, chair of the Royal College of GPs (RCGP), told Digital Health News that it has concerns “around the potential patient safety risks associated with current AI-assisted tools” .

“We would like to see clear evidence from an independent and unbiased clinical safety evaluation of any AI-assisted discharge summary tool, demonstrating that it is at least as safe and effective as current discharge summary processes before it is rolled out more widely.

“This evaluation should include real-world testing across diverse clinical settings, including high-pressure environments, and full publication of the findings,” Brown said.

RCGP is a member of the FDP Check and Challenge Group, which was given a demonstration of the discharge summary product in its September 2025 meeting.

At the meeting, members questioned whether the tool had been classified as a medical device by the Medicines and Healthcare products Regulatory Agency (MHRA) and raised the risk of loss of clinically relevant information which could affect ongoing care and safety.

A member also highlighted that the use of patient data to validate and re-validate a tool under development does not clearly fall within the scope of direct care.

The group cautioned that the tool needed “further clinical depth and regulatory assurance, and called for visibility of further evidence of safety, clinical appropriateness, and formal regulation before further deployment,” according to meeting minutes.

In response, the FDP team confirmed that discussions with the MHRA regarding medical device classification were ongoing and that the tool was designed to support clinical judgment, with multiple validation steps built in to mitigate risks of information loss.

A spokesperson for the British Medical Association told Digital Health News: “Before allowing new technology such as this to take a central role in handling patient data, we must first understand its impact on patients, doctors and the system as a whole.

“​In our meetings with the check and challenge group, we have been assured that human sign-off will be required as the final step, but without fully exploring the potential risks, we remain concerned about the speed at which this new technology is being implemented.”

Anna Steere, head of Understanding Data, told Digital Health News that with the right safeguards in place the discharge tool could be  “a pragmatic step forward in a system we already know needs significant improvement”.

She added that “if clinicians over-rely on AI and stop checking outputs carefully, mistakes could slip through”, but said this was about setting clear rules for the use of AI across the health service as a whole.

Digital Health News contacted Chelsea and Westminster NHS Trust and NHS England for comment.