Clinical AI advances outpace safety evidence
Evidence supporting the safe and effective use of clinical artificial intelligence in routine care has not kept pace with advances in model capabilities, according to a narrative review.
The review, published in BMJ Digital Health & AI, cited evidence that fewer than 5% of more than 1,200 FDA-cleared medical devices using artificial intelligence (AI), machine learning, or both had undergone rigorous peer-reviewed evaluation. Prospective randomized trials also remained limited.
“Across domains, a consistent theme emerges: model capabilities have progressed faster than the research and implementation systems needed for safe deployment,” the authors wrote.
The review synthesized findings from 82 studies examining 6 areas of clinical AI: clinical workflows, patient-facing applications, model performance, benchmarks and evaluation, foundational methods, and applied AI. It drew primarily on the AI Research and Science Evaluation (ARISE) network’s State of Clinical AI Report 2026.
Researchers conducted a targeted search of PubMed, medRxiv, and arXiv from October 2025 through January 2026, supplemented by expert nominations. Of the 82 included studies, 40 were identified through database searches, 31 were nominated by the author team, and 11 were submitted by external clinicians and AI researchers. The authors used a Delphi consensus process to select nominated studies and establish the 6 domains.
1. Clinical workflows
Early studies suggested improved diagnostic accuracy and reduced administrative burden, but evidence of patient benefit remained limited.
A large-scale quality improvement study conducted across 15 clinics in Kenya evaluated clinical decision support based on large language models (LLMs) during 39,849 patient visits. The intervention was associated with relative reductions of 16% in physician-rated diagnostic errors and 13% in treatment errors. Patient-reported outcomes, however, did not differ significantly. The review noted that further studies were needed to assess clinical outcomes and generalizability, given the system’s reliance on extensive prompts and contextual information.
In a randomized trial involving 70 physicians, diagnostic accuracy was 75% without AI assistance, 82% when AI provided a second opinion, and 85% when it provided an initial opinion. However, the combined physician-AI approaches did not significantly outperform AI alone, which achieved 87% accuracy.
A meta-analysis of 52 studies similarly found better diagnostic reliability with physician-AI collaboration than with physicians alone, but collaboration rarely outperformed both physicians and AI working separately.
The authors cautioned that, in some studies, physicians lacked access to clinical decision-support tools they would ordinarily use in practice, which could overstate AI’s relative advantage.
Studies of ambient AI scribes reported reductions in perceived cognitive workload and documentation burden. However, some reported time savings of approximately 20 seconds per note, and clinically meaningful documentation errors remained a concern.
The review also identified risks associated with automation bias and potential deterioration of clinical skills. It reported that, in a randomized trial, physicians exposed to erroneous AI recommendations scored 14 percentage points lower in diagnostic accuracy than those in the control group.
2. Patient-facing applications
Patient-facing applications demonstrated early evidence of benefits in health behavior support, including diabetes prevention.
The review described a randomized controlled trial involving 368 adults in which an AI-powered diabetes prevention program was noninferior to human coaching. Assessed outcomes included body weight, hemoglobin A1c, and physical activity.
The review also described a personalized health language model that generated sleep and fitness recommendations at or above the level of human experts in evaluated tasks and outperformed its general-purpose counterparts.
Despite these findings, the authors cautioned that patient preference and perceived response quality are unreliable indicators of medical safety.
Studies showed that patients could not consistently distinguish AI-generated medical advice from clinician-authored responses and sometimes rated inaccurate recommendations as equally or more trustworthy. Participants also reported comparable willingness to follow potentially harmful, low-accuracy advice and higher-accuracy recommendations.
The authors emphasized the need for safety evaluation and safeguards that do not depend on patients recognizing inaccurate information.
3. Model performance
LLMs demonstrated substantial improvements in diagnostic reasoning and disease management tasks compared with earlier systems and, in some controlled evaluations, physicians.
In one evaluation using historically difficult New England Journal of Medicine clinicopathological conference cases, OpenAI's o3 model achieved 60% top-choice diagnostic accuracy and included the correct diagnosis among its top 10 possibilities in 84% of cases. It selected the correct next diagnostic test 98% of the time.
The review also reported that physicians with unrestricted electronic search achieved 24% top-choice accuracy and 45% accuracy when considering their top 10 diagnoses in an evaluation involving 302 clinicopathological conference cases.
Specialized systems also demonstrated improvements in clinical prediction. AgentMD, which autonomously selects from more than 2,000 clinical calculators, achieved 88% accuracy on the 350-question RiskQA benchmark, compared with 41% for general-purpose models.
In an application relevant to endocrinology, the review described research combining wearable data and routine blood biomarkers to predict insulin resistance.
However, performance declined when models encountered unfamiliar or more complex scenarios. In one study, replacing the correct answer in multiple-choice questions with a “None of the other answers” option reduced model performance by 9% to 38%.
Despite these gains, models struggled to gauge their own uncertainty, recognize knowledge limitations, and maintain performance when patient populations or care settings differed from their training data.
4. Benchmarks and evaluation
The authors found that conventional medical knowledge benchmarks frequently failed to reflect the demands of clinical practice.
A systematic review of 519 studies published between 2022 and 2024 found that 95% used accuracy as the primary evaluation metric. Medical licensing examination questions were evaluated in 45% of studies, while only 5% incorporated real patient data.
Newer evaluation frameworks increasingly incorporate clinical conversations, longitudinal decision-making, patient safety, and administrative workflows. For example, Medical Holistic Evaluation of Large Language Models (MedHELM) assessed models across 121 distinct clinical tasks, while MedAgentBench evaluated their ability to perform 300 patient-specific tasks in a simulated electronic health record environment.
Despite improvements in medical reasoning, models performed relatively poorly on complex, multistep clinical tasks involving electronic health records, interpretation of user requests, and long-term planning.
In the Numerous Options Harm Assessment for Risk in Medicine (NOHARM) benchmark, frontier models generated potentially harmful recommendations in approximately 22% of 100 primary care–to-specialist consultation cases across 10 specialties.
These errors frequently involved omissions of diagnostic tests or treatments considered necessary for patient safety. The authors also cautioned that automated evaluations using other language models as judges could introduce potential reliability and bias concerns.
5. Foundational methods
Foundation models trained on longitudinal medical records demonstrated expanding capabilities in disease prediction and risk stratification.
Delphi-2M, a medical event foundation model, achieved an average area under the curve of 0.76 when predicting the occurrence and timing of more than 1,000 diseases. The model could simulate health trajectories extending up to 20 years.
However, the authors emphasized that discrimination alone is insufficient for clinical decision-making and that these predictions require external validation and prospective studies. They also cautioned that the predicted patterns should not be interpreted as causal relationships and that individual predictions remain substantially influenced by chance.
Multimodal systems integrating clinical notes, laboratory results, and medical imaging also demonstrated improvements in selected diagnostic and prognostic tasks. Nevertheless, the review identified instances in which these models generated confident explanations for incorrect diagnoses, including descriptions of imaging features that were not present.
Multiagent systems presented additional challenges. In one evaluation involving 2,400 emergency department cases, a system assembled from individually high-performing agents achieved 86% component-level accuracy but only 68% diagnostic accuracy, compared with 77% for a less-optimized system.
The authors attributed this discrepancy to failures in information exchange between agents, illustrating that improvements in individual model performance do not necessarily improve the complete system.
6. Applied AI
The review identified narrowly defined clinical tasks with measurable outcomes as the areas demonstrating the most credible progress toward clinical implementation. Imaging remained the dominant application, with advances reported across pathology, neurology, nephrology, oncology, pulmonology, and cardiology.
In a prospective observational study across 26 hospitals in England, patients whose stroke imaging was reviewed with AI had higher odds of good functional outcomes. These observational findings did not establish that AI caused the improvement.
Limitations and future priorities
The authors acknowledged several limitations. The review used a targeted literature search and expert nominations rather than a systematic review with predefined search protocols.
Expert nominations may have introduced selection bias toward studies from well-connected academic institutions. The findings also represented a snapshot of a rapidly evolving field in which preprints may change during peer review and new models may alter comparative performance.
The review also did not examine equity, algorithmic bias, regulation, governance, or implementation science in depth. Few included studies reported outcomes stratified by population, limiting assessment of performance differences across demographic groups.
The authors called for future research to include prospective multisite trials embedded in routine clinical workflows that measure patient outcomes, workflow effects, and safety events. They also recommended evaluation beyond accuracy to assess uncertainty, equity, and longitudinal performance. A third priority was patient and user safety, including human-AI interaction that reduces automation bias and preserves physician expertise.
They added that advances in clinical AI have not yet established the evidence needed for broader clinical autonomy. “The gap between what clinical AI can do in controlled settings and what it can be trusted to do at the bedside is one of the defining tensions of this new field,” the authors wrote.
A list of the authors' financial disclosures can be found in the review's Competing interests section.
AACE Endocrine AI is published by Conexiant under a license arrangement with the American Association of Clinical Endocrinology, Inc. (AACE®). The ideas and opinions expressed in AACE Endocrine AI do not necessarily reflect those of Conexiant or AACE. For more information, see Policies.