AI scribes boost documentation, but at a cost
Artificial intelligence (AI) scribes were linked to more frequent documentation of patient instructions during outpatient visits, but the instructions were longer and harder to read than those written by physicians without AI assistance, according to a study of nearly 400,000 patient encounters.
The findings, published in the Journal of the American Medical Informatics Association, suggest that AI scribes may help ensure that more patients receive written instructions, although those instructions may be harder to understand.
"It isn't entirely clear that increased length and increased reading complexity necessarily translates to less clarity or utility for patients, though this is a reasonable assumption," first author Kevin Shi, MD, of the Division of Nephrology at the University of California, San Francisco, told this news organization. "At this time, it is difficult to confidently increase instruction quality without increasing physician burden."
Dr. Shi and colleagues reviewed records from 391,061 outpatient visits involving adults between December 2023 and July 2025. The study included 399 physicians across 8 specialties, including endocrinology, cardiology, gastroenterology, nephrology, and primary care.
The researchers examined whether access to an AI scribe was linked to more frequent inclusion of patient instructions in after-visit summaries. They also compared the length and readability of instructions generated by AI with those written by physicians.
Of the 391,061 visits, 61% included written patient instructions, and AI scribes were used in 13%. Of the 399 physicians, 146 began using AI scribes, whereas 253 did not.
Documentation increased
The researchers compared how often physicians documented patient instructions before and after adopting AI scribes, using physicians who did not adopt the technology as a comparison group. They found that access to AI scribes was linked to a 10.5-percentage-point increase in the frequency of documented instructions.
Among physicians who adopted AI scribes, the percentage of visits with documented patient instructions increased from 68% to 79%. Among those who did not adopt the technology, the rate remained at 68%.
The greatest improvement was seen in primary care, where documentation increased by 16.6% compared with physicians who did not use AI scribes. Across other specialties, the increase was 4.9%.
After accounting for differences in patient, physician, and visit characteristics, the researchers found that AI scribe use was linked to more than 4 times the odds of documenting patient instructions.
Overall, patient instructions were documented in 86% of visits involving AI scribes, compared with 58% of visits without them.
Patient instructions were more likely to be documented during initial visits and visits with English-speaking patients. Documentation was less common during telehealth visits and among more experienced physicians.
Longer, more complex instructions
Although AI scribes were linked to more frequent documentation, AI-generated instructions were about 4 times longer than those written by physicians, with a median of 1,994 characters vs 482 characters.
AI-generated instructions were also harder to read, according to 3 standard measures of reading difficulty. On the Flesch-Kincaid scale, they required an average reading level equivalent to grade 16.5, compared with grade 12 for instructions written by physicians.
Physician review did little to improve readability. AI-generated instructions edited by physicians averaged 2,093 characters and had a Flesch-Kincaid reading grade level of 15.9.
Among 43,155 visits with both AI-generated drafts and final instructions, about 40% of the text differed between versions, suggesting that physicians made substantial edits.
The researchers noted that all 3 types of instructions were harder to read than the grade 6 reading level commonly recommended for patient materials.
Clinical implications and limitations
The findings suggest that physicians who use AI scribes should pay close attention to the length and readability of patient instructions because reviewing AI-generated text may not make it easier for patients to understand.
Study limitations included its retrospective design, use of data from a single medical center, and the possibility that physicians were more likely to use AI scribes during complex visits. The readability scores also could not show whether patients understood the instructions.
The researchers also did not directly evaluate the accuracy of the instructions, patient satisfaction, whether patients followed the instructions, or patient health outcomes.
“As AI documentation technologies continue to evolve, careful evaluation of their effects on clinical workflows, patient communication, and patient outcomes will be essential to ensure that gains in documentation efficiency translate into meaningful improvements in care,” the authors concluded.
Dr. Shi received support from the National Institute of Diabetes and Digestive and Kidney Diseases. The authors reported no competing financial interests or personal relationships that influenced the study.
Dr. Shi provided additional context on the study findings.
Your study found that AI scribes increased the documentation of patient instructions but also made them longer and more complex. What should endocrinologists do to ensure these instructions are clear and useful to patients?
Dr. Shi: The lowest hanging fruit would likely be to change the framing of instructions. For example, ensuring patients are aware that there actually are personalized instructions for them to review and restructuring After Visit Summaries or clinic printouts to emphasize these instructions (e.g., make sure they are on the first page of paperwork).
For patients managing chronic conditions such as diabetes, what are the potential risks of relying on AI-generated instructions for medications, glucose monitoring, and follow-up care?
Dr. Shi: Ambient AI and large language models are increasingly reliable when it comes to generating clinically appropriate advice. Instances of overtly harmful hallucinations are becoming rarer, and these problems are generally easier to detect on clinical review. The presence of omissions, or clinically appropriate advice that should have been included but wasn't, is more challenging to detect and rectify, especially as clinician trust in AI continues to grow. Building clinical diligence, especially when it comes to omissions, will be an important skill for providers as roles shift from being primary documentarians to more editing and supervisory roles. When it comes to common endocrine conditions like diabetes, specific attention should be on logistic assumptions (e.g., the AI may assume that the patient has a glucometer) and personalized goals (e.g., glucose or A1c goals that are informed by holistic health factors that an AI scribe may not be able to detect).
What safeguards should endocrinology practices put in place when using AI scribes, and what further research is needed to determine whether these tools improve patient understanding and treatment adherence?
Dr. Shi: Endocrine disorders are heavily affected by lifestyle and self-management. Patient communication and patient instructions are extremely important in this arena. As things stand, there is little data to support imposing any clear safeguards on AI scribe use, but monitoring and stewardship are paramount to clinical success. Strategies like monitoring overall AI utilization, the amount and character of physician editing of AI content, and patient-level outcomes (including satisfaction and other non-clinical outcomes) are useful to ensure successful implementation.
AACE Endocrine AI is published by Conexiant under a license arrangement with the American Association of Clinical Endocrinology, Inc. (AACE®). The ideas and opinions expressed in AACE Endocrine AI do not necessarily reflect those of Conexiant or AACE. For more information, see Policies.