Abstract / Summary
Aggregate large language model (LLM) performance may obscure consequential errors and variability across repeated queries. We compared the clinical performance and safety of three consumer-facing LLMs with two fixed individual early-career urologist comparators in urological oncology and separately assessed repeated-query response stability in the LLMs. One hundred synthetic vignettes spanning five urological cancers and five decision categories were answered by GPT-5.2 Instant, Claude Sonnet 4.5, Gemini 3 Pro Preview, and two early-career urologists using the same standardized prompt without external resources. Two source-masked urologists with uro-oncology expertise independently scored all 500 initial responses against a prespecified 2025 European Association of Urology (EAU)-anchored framework. Benchmark-defined primary composite success required scores ≥ 2 in guideline concordance, clinical safety, decision completeness, and risk/stage accuracy, with no prespecified disqualifying clinical error. Safety–critical errors were assessed across all five response sources, whereas exact three-run response stability was assessed separately in the three LLMs. Primary composite success was achieved in 70/100 (70.0%) GPT-5.2, 64/100 (64.0%) Claude, and 60/100 (60.0%) Gemini responses, versus 85/100 (85.0%) and 89/100 (89.0%) for Urologists A and B (Cochran’s Q = 34.23; P < 0.001). Five of six prespecified human–LLM contrasts remained significant after Holm adjustment; Urologist A versus GPT-5.2 did not (difference, 15 percentage points; 95% CI, 3–27; adjusted P = 0.095). Safety–critical errors occurred in 25.0%, 25.0%, and 26.0% of LLM responses versus 2.0% and 1.0% of urologist responses (Cochran’s Q = 51.34; P < 0.001); all six human–LLM contrasts remained significant. Across 60 LLM vignette–platform combinations, exact three-run stability was 45.0% for primary classification, 48.3% for safety–critical status, and 28.3% for the full four-domain score profile. Stable primary classifications included both consistently successful and consistently unsuccessful outputs; stable safety–critical status included both absence and presence of a safety–critical error across all three runs. The consumer-facing LLMs evaluated during the study period often produced acceptable responses, but safety–critical errors and limited repeatability remained clinically relevant. Aggregate performance did not fully capture response reliability. The findings support evaluating performance, consequential error, and repeatability separately before clinical deployment. Whether clinician supervision mitigates the observed failure modes requires direct evaluation in prospective workflow studies.