The interesting thing about large language models in medicine is that the argument about whether to use them is largely over. Ambient scribes are already writing consultation notes. Triage chatbots already sit on hospital websites. Discharge summaries, prior authorisation letters, radiology report drafts and patient messaging replies are already being generated and then, in theory, checked.
The argument that remains is about what happens when one of these systems is wrong, manipulated, or quietly biased, and nobody notices for six months.
That question has a different weight in health than it does anywhere else. A hallucinated figure in a marketing deck is embarrassing. A hallucinated figure in a medication reconciliation is a patient safety incident.
The Failure Modes Are Not The Ones People Prepare For
Most health organisations preparing for AI risk are thinking about accuracy. Accuracy is the least interesting problem, because it is the one everybody is already watching.
The failures that reach patients tend to come from somewhere else.
Patient Data That Comes Back Out
When a model is fine-tuned or given retrieval access to clinical records, correspondence or internal documentation, the boundary between what it knows and what it is allowed to say becomes a matter of configuration rather than physics.
The obvious version, a model reciting a patient identifier verbatim, is not the one that usually happens. The common version is subtler. A model that has been trained or grounded on a small speciality caseload can produce a response that is specific enough to re-identify someone even without naming them. A rare condition, a district, an age, a treatment sequence. Four attributes that individually reveal nothing and jointly reveal a person.
Access control at the database layer does not solve this, because the model is not querying the database at the moment it answers. It is drawing on whatever was folded into its weights or pulled into its context window. Controls have to sit at four points: what goes into training or fine-tuning, what the retrieval layer is permitted to fetch for a given user, what the model is allowed to say, and what gets logged so that a leak can be detected after the fact rather than discovered by a journalist.
Bias In Health Data Is Documented, Not Hypothetical
The reason to take algorithmic bias seriously in clinical settings is not that it might happen. It is that it has been measured.
The landmark study here predates the current generation of models but explains the mechanism better than anything since. Researchers examining a risk prediction algorithm used across the US health system found that at any given risk score, Black patients were considerably sicker than White patients with the same score. The algorithm was not using race as an input. It was predicting healthcare costs as a proxy for healthcare need, and because less money had historically been spent on Black patients with equivalent illness, the model concluded they were healthier. Correcting the target variable raised the proportion of Black patients flagged for additional care from 17.7% to 46.5%.
Nothing about that failure was visible from accuracy metrics. The model was performing well against the thing it had been asked to predict. The thing it had been asked to predict was wrong.
Language models inherit this problem and add to it, because the proxies are now buried inside text rather than sitting in a labelled column where an analyst might spot them.
Prompt Injection Is A Clinical Safety Issue, Not An IT One
This is the risk most health organisations are least prepared for, and the evidence is now uncomfortable.
A study published in Nature Communications tested whether vision-language models used in oncology could be compromised through prompt injection. Across 594 attacks against four leading models, all four were susceptible. The researchers embedded sub-visual instructions directly into medical imaging data, and the instructions were not obvious to human observers. A clinician reviewing the same image would see nothing unusual while the model followed hidden instructions.
Sit with the implication. The attack surface is not the chat box. It is anything the model reads: a scanned referral letter, a patient-uploaded photograph, a PDF from an outside laboratory, a fax that was OCR’d into the record. Any of those can carry instructions that the model treats as authoritative.
A subsequent study in JAMA Network Open examined whether commercial medical LLMs could be pushed into recommending unsafe or contraindicated treatments through injected content across a controlled set of patient dialogues. The mechanism is the same, and the output is a treatment recommendation rather than a leaked file.
The critical point for governance committees is that this is not a data breach in the traditional sense. Nothing is stolen. The system does exactly what it was asked to do by whoever got their instructions in last.
The Failure That Requires A Human To Complete It
There is a category of risk that no amount of model hardening addresses, because the vulnerability is in the workflow rather than the software.
A randomised clinical trial published in JAMA Network Open assigned physicians to work through diagnostic cases either with conventional resources or with conventional resources plus an LLM. The result was not what most people expect: access to the language model did not significantly improve diagnostic performance compared with conventional resources alone, even though the model performed strongly when evaluated on its own.
That gap between what a model can do in isolation and what it does when a clinician is in the loop is the whole ballgame. Deployment decisions are usually justified with benchmark scores. Benchmark scores describe the model. They do not describe the sociotechnical system that includes a tired registrar at 3am who has learnt over six months that the tool is usually right.
Automation bias is the term for what happens next. It is a documented human factors problem, and it gets worse, not better, as a system becomes more reliable, because reliability is what teaches people to stop checking.
Why A Security Review In Healthcare Is Not The Same Review
Standard application security testing asks whether an attacker can get in. That question is necessary and insufficient here, because most of the failure modes above involve no unauthorised access at all.
Testing an LLM deployment in a clinical environment means asking a different set of questions. Can content inside a document the system is designed to read change its behaviour? Can a user reach information belonging to a different patient through ordinary conversation rather than exploitation? Does the model treat retrieved content as data or as instruction? What happens at the boundary where the model can call a tool, write to a record, or trigger an order?
Automated scanners handle the first layer. They do not handle multi-step, semantically creative attacks, because those require somebody to think like an adversary who has read the clinical workflow. This is where a structured LLM security audit conducted by people who test adversarially, rather than a compliance questionnaire completed internally, earns its cost.
The World Health Organization reached a similar conclusion. Its guidance on large multi-modal models in health, published in January 2024, sets out more than forty recommendations and explicitly calls on governments to require post-release audits and impact assessments conducted by independent third parties. Not pre-launch. Post-release, continuing.
The reasoning is straightforward. Models get updated by vendors without notice. Integrations accumulate. Clinical use drifts away from the use that was originally approved. A security posture established at go-live describes a system that no longer exists.
For teams building an internal testing programme, the OWASP Top 10 for LLM Applications is the most widely used starting taxonomy, and it maps reasonably well onto clinical deployments even though it was not written for them.
What Regulators Now Expect
Health organisations sit under more overlapping obligations than most sectors, and generative AI touches several at once.
Data protection law applies wherever identifiable patient information reaches a model, which includes information typed into a general-purpose consumer chatbot by a well-meaning clinician. That is a disclosure to a third party, regardless of intent.
Medical device regulation is the harder question. Where a system influences diagnosis or treatment, it may fall under device rules rather than general software rules, which brings clinical evaluation, post-market surveillance and change control obligations. Vendors are frequently vague about which side of that line their product sits on, and the deploying organisation carries real exposure if the answer is wrong.
The EU AI Act adds a further layer for anything operating in or into the European market, with obligations scaled to risk classification. Health applications frequently land in the higher tiers.
An international consensus guideline, FUTURE-AI, sets out principles for trustworthy and deployable AI in healthcare across the full lifecycle, and is a more practical reference for clinical teams than most regulatory text. A systematic review of the ethics of large language models in medicine and healthcare is useful for governance committees that need the evidence base rather than the vendor summary.
What Actually Reduces Risk
Nothing in this list is exotic. The difficulty is organisational rather than technical.
Restrict retrieval by user and session rather than by system. The model should be able to reach only what the person in front of it is entitled to see, enforced at query time.
Treat all retrieved content as untrusted input. Content inside a document is data, never instruction, and the architecture should enforce that rather than relying on the model to be sensible about it.
Log inputs and outputs at a level that supports reconstruction. When something goes wrong, the question will be which prompt produced which output for which patient, and if that cannot be answered the incident cannot be closed.
Define the human review step precisely, then check that it is happening. A review step that exists in the policy and not in the workflow is worse than no review step, because it creates the impression of a control.
Re-test after every model update, integration change or expansion of scope. Vendor-side changes count.
Train clinical staff on what these systems fail at, not just how to use them. The person best placed to catch a hallucinated dose is the one who was told to expect one.
Questions Health IT Teams Should Be Asking
Can a patient’s data leak without a breach? Yes. Retrieval misconfiguration and over-broad fine-tuning both produce disclosure without any unauthorised access occurring.
Does our vendor’s certification cover this? Usually not. SOC 2 and ISO 27001 address organisational security controls. Neither evaluates whether a model can be talked into a contraindicated recommendation.
How often should we re-test? After every significant model or integration change, at minimum. Clinical deployments touching diagnosis or medication warrant a shorter cycle regardless of change activity.
Is a general-purpose chatbot ever acceptable for clinical use? Not with identifiable patient data, absent a specific agreement covering data handling, retention and residency. The default terms of most consumer products are incompatible with clinical confidentiality obligations.
Who is accountable when the output is wrong? In practice, the clinician who acted on it and the organisation that deployed it. Vendor contracts rarely transfer clinical liability, and this is worth reading carefully before signing.
The Bottom Line
The risks described here are not speculative. They have been measured in peer-reviewed studies, using the models that health organisations are currently deploying, on tasks those models are currently being asked to perform.
What makes healthcare distinct is not that the technology is riskier here. It is that the error tolerance is lower, the affected party has no visibility into the system, and the harm may not be discovered for a long time.
The organisations that will handle this well are not the ones that move slowest. They are the ones that treat an AI deployment as a clinical system requiring surveillance rather than an IT project requiring sign-off.
Disclaimer
This article is intended for general information and professional awareness. It does not constitute medical, legal, or regulatory advice, and it is not a substitute for guidance from qualified clinical, legal, information governance, or cybersecurity professionals.
Nothing here should be read as clinical guidance for patients. Readers should not use large language models as a substitute for consultation with a qualified healthcare professional, and should not act on AI-generated health information without professional review.
Regulatory requirements affecting AI in healthcare, including data protection law, medical device classification and AI-specific legislation, vary considerably between jurisdictions and are changing rapidly. Organisations should obtain advice specific to their jurisdiction, their deployment, and their data.
References to specific studies describe findings as published. They are not endorsements of any product, vendor, or service, and no commercial relationship exists between this article and the systems evaluated in the cited research.
References
- Obermeyer Z, Powers B, Vogeli C, Mullainathan S. Dissecting racial bias in an algorithm used to manage the health of populations. Science. 2019;366(6464):447-453. doi:10.1126/science.aax2342
- Clusmann J, Ferber D, Wiest IC, Schneider CV, Brinker TJ, Foersch S, Truhn D, Kather JN. Prompt injection attacks on vision language models in oncology. Nature Communications. 2025;16(1):1239. doi:10.1038/s41467-024-55631-x
- Lee RW, Jun TJ, Lee JM, Cho SI, Park HJ, Suh J. Vulnerability of large language models to prompt injection when providing medical advice. JAMA Network Open. 2025;8(12):e2549963. doi:10.1001/jamanetworkopen.2025.49963
- Goh E, Gallo R, Hom J, et al. Large language model influence on diagnostic reasoning: a randomized clinical trial. JAMA Network Open. 2024;7(10):e2440969. doi:10.1001/jamanetworkopen.2024.40969
- Haltaufderheide J, Ranisch R. The ethics of ChatGPT in medicine and healthcare: a systematic review on large language models (LLMs). npj Digital Medicine. 2024;7(1):183. doi:10.1038/s41746-024-01157-x
- Lekadir K, Frangi AF, Porras AR, et al. FUTURE-AI: international consensus guideline for trustworthy and deployable artificial intelligence in healthcare. BMJ. 2025;388:e081554. doi:10.1136/bmj-2024-081554
- Singhal K, Azizi S, Tu T, et al. Large language models encode clinical knowledge. Nature. 2023;620(7972):172-180. doi:10.1038/s41586-023-06291-2
- Savage T, Nayak A, Gallo R, Rangan E, Chen JH. Diagnostic reasoning prompts reveal the potential for large language model interpretability in medicine. npj Digital Medicine. 2024;7(1):20. doi:10.1038/s41746-024-01010-1
- Sonicki Z. Large multi-modal models: the present or future of artificial intelligence in medicine? Croatian Medical Journal. 2024;65(1):1-2. doi:10.3325/cmj.2024.65.1
- World Health Organization. Ethics and Governance of Artificial Intelligence for Health: Guidance on Large Multi-Modal Models. Geneva: World Health Organization; 2024. ISBN 978-92-4-008475-9.