Clinician-centered evaluation of large language model-generated discharge summaries for longer hospitalizations

Although large language models (LLMs) have shown promise for discharge summary generation, their performance in longer hospitalizations remains less well studied. Prior evaluations of LLM-generated discharge summaries have largely involved shorter stays and have rarely examined receiving-clinician priorities or incidental finding reporting. We compared LLM-generated and human-authored discharge summaries for 60 Internal Medicine hospitalizations lasting 7 to 21 days, with paired assessment by hospitalists and primary care physicians (PCPs). Clinician reviewers preferred LLM-generated summaries for 95% of encounters and rated them higher for quality, readability, factuality and completeness. PCPs rated LLM-generated summaries as better for understanding and communicating hospital care to patients and providing follow-up care. LLM-generated summaries had fewer clinician-reviewer annotations of either factual inaccuracies or clinically relevant omissions, primarily due to fewer omissions, with no significant differences in estimated harm potential or likelihood compared with human-authored summaries. PCPs annotated more omissions and assigned higher likelihood of harm than hospitalists. Among 31 LLM-identified radiology incidental findings, 93.5% were rated factually correct and 87.1% appropriate for reporting. In this retrospective clinician evaluation, LLM-generated summaries for longer, complex hospitalizations were rated more favorably and had fewer annotated omissions than human-authored summaries, with no significant differences in estimated harm potential or likelihood.