Performance evaluation and benchmarking across 16 large language model configurations on a large real-world emergency department triage data set

Background Emergency department (ED) triage is a decision process that determines patient prioritization and resource allocation. Large language models (LLMs) have been proposed as decision-support tools for triage, yet most evaluations rely on curated datasets. Evidence from real-world clinical environments remains limited. The objective of this project was to evaluate the performance and reproducibility of multiple contemporary large language models for Emergency Severity Index (ESI) classification and sectoral allocation (ED vs. urgent care practice, UCP) using a real-world triage dataset. Material and Methods Retrospective cross-sectional benchmarking study conducted at a tertiary academic ED in Germany with an integrated central point of assessment (CPA). The study included all 16,107 walk-in encounters between October 18, 2023, and February 27, 2024. Sixteen configurations were compared on the first 2,000 encounters; GPT 5.1 with high reasoning effort was additionally evaluated on all 16,107 encounters. Data were collected from a structured clinical decision support system capturing presenting complaints, vital signs, and triage decisions recorded by specialized nursing staff. Results The primary outcome was the agreement between LLM-predicted and nurse-assigned ESI levels measured using quadratic-weighted Cohen's kappa. Secondary outcomes included sectoral assignment agreement, misclassification patterns (over- and under-triage), confidence-agreement comparisons, and output reproducibility. In the primary analysis of all 16,107 encounters, GPT 5.1 with high reasoning effort achieved quadratic-weighted kappa = 0.5031 (95% CI 0.4904-0.5158). In the exploratory comparison of 16 configurations on the shared 2,000-encounter subset, quadratic-weighted kappa ranged from 0.180 to 0.860. The Stepwise approach, combining structured model assessments with deterministic ESI mapping, achieved quadratic-weighted kappa = 0.860 (95% CI 0.841-0.878) and 82.70% exact agreement. Most models demonstrated moderate or lower agreement. Sectoral assignment agreement (i.e. ED vs. urgent care practice, UCP) was uniformly low (kappa of 0.304 or lower). Reproducibility testing revealed substantial variability in 23% of cases, indicating non-deterministic output behavior for clinically relevant decisions. Conclusions Current large language models demonstrate heterogeneous and generally limited performance in real-world emergency triage tasks. The Stepwise approach achieved the highest agreement. Before clinical implementation, confidence estimates, reliability, and workflow integration require prospective validation in regulatory-compliant clinical settings.