Why Healthcare May Be the Hardest Test for Sovereign AI

By Henry Duah

The consultation room looks much as it always did. A clinician, a patient, perhaps a relative in the corner chair. What is missing: the clinician is not typing. A device listens, and by the time the patient has left, a structured note is waiting to be reviewed. Of all the ways AI has entered healthcare, ambient scribes may be the least dramatic and the most widespread. They do not read scans or recommend treatments. They write things down. That modesty is why they scaled first, and it is why the questions they raise are less about clinical brilliance than about governance.

What an AI scribe actually does

An ambient AI scribe combines speech recognition with generative AI to turn a conversation into a transcript and a summary. Depending on the product, that summary can follow a template, become a referral or patient letter, and be written straight into the health record. NHS England’s guidance describes the latest generation as pairing advanced speech technology with large language models, and stresses that users must review and approve the output before anything further happens. The tool used at some hospitals does not make decisions or recommendations about patient care. Patients are told it is in use, and doctors can edit the summary before it is filed.

Real relief, modest time saving

Research by academics and AI scribe suppliers has been ongoing for some time now. One of the largest published rollouts of AI Scribe comes from a healthcare group in Northern California, USA. The study shows that between October 2023 and December 2024, 7,260 physicians used AI scribe in over 2.5 million patient encounters, and the analysis estimated that it saved nearly 16,000 hours of documentation time. In another survey of 102 physicians, 88 per cent reported a positive effect on their interactions during visits. Of 118 patients surveyed, 47 per cent thought their doctor spent less time looking at the computer. Another five-centre study, as reported by the Textbook of Digital Health and STAT, found savings of around 16 minutes per 8 hours of patient care, rising to about 27 minutes for clinicians who used an AI scribe in at least half of their visits.

Put together, the evidence suggests genuine relief for clinicians, modest time savings, and an unproven business case. That is a respectable start. It is not yet a reason for anyone to stop asking what could go wrong.

The risk that sits in the draft

An AI scribe’s output looks like a finished clinical note, and that is precisely the hazard. NHS England warns of errors and omissions, automation bias among users who stop checking, and contextual misunderstandings, such as mistaking a patient’s hypothetical statement for a confirmed diagnosis. It adds that speech products may perform unevenly with certain accents, regional dialects, or speech impairments, and asks adopters to monitor whether any patient groups hesitate to seek care because a scribe is present.

Some earlier pilot scored a sample of 35 AI-generated transcripts and found few hallucinations. That is encouraging, but 35 transcripts is a pilot, not a safety case. Beyond accuracy, the regulatory status is subtle. NHS England says tools that only produce transcripts are unlikely to be medical devices, while those that use generative AI to summarise are likely to be. Its supplier registry requires at least registration with the UK medicines regulator, the Medicines and Healthcare products Regulatory Agency, as a Class 1 device, along with clinical-safety documentation and a data protection impact assessment from the adopting organisation. 

On liability, the guidance is blunt: a hospital or practice may still carry a non-delegable duty of care, even when a supplier’s product made the error. If a scribe’s mistake harms someone and no other party can clearly be held liable, responsibility can fall back on the hospital or practice, which may then seek redress from the supplier.

The tools nobody approved

A December 2025 survey of 518 healthcare workers by Wolters Kluwer Health, half providers and half administrators, found that 17 per cent admitted to using unauthorised AI tools at work. Among those who had, 45 per cent said they did so for a faster workflow, and a further 24 per cent because the tool had better functionality than the approved options. It would be easy to assume this is a problem of junior staff. The survey suggests otherwise: among providers with fewer than five years’ experience, only 14 per cent had used unsanctioned tools. One caution is that Wolters Kluwer sells clinical decision-support software, so the survey’s framing deserves a critical read, though its core numbers are plausible. 

NHS England’s guidance reaches a similar conclusion from the regulator’s side. It acknowledges that some users may use products outside the supervision of their setting, circumventing policies and approvals. That is the governance gap. Every safeguard – the safety case, the impact assessment, the independent audit, the notice to patients – attaches to a tool the organisation knows about. A tool a clinician quietly adopts has none of it, yet it may still record the same conversations and produce the same notes.

Closing the gap

Survey data from published research suggest where to start. If a quarter of unauthorised users say the approved tools were not good enough, then the first control is a sanctioned option that clinicians actually prefer, and that was chosen with their input. Some suppliers’ own recommendation is the same: involve providers in tool selection, or they will return to their preferred tools in the shadows.

The second is to treat review as work rather than a formality. NHS England asks for training on dictation and on the responsibility to check outputs, and for monitoring data that is analysed independently of the supplier’s.

The third is honesty with patients about what is recorded, who sees the output and how it is stored. And pilots need an end date: the guidance allows them for up to four months, and only if they are not used to bypass compliance.

The ambient AI scribe is a small piece of software doing a humble job. But it sits at the point where a private conversation becomes a legal record, making it a revealing test of whether health systems can govern AI as quickly as their clinicians adopt it. The technology is already in the room. The question is whether the rules are.

Get innovation insights from The FutureList weekly. Subscribe to our newsletter here