Evaluating medical Large Language Models (LLMs) for clinical decision support: real-world performance across specialties and complexity levels

Large Language Models (LLMs) show increasing promise for clinical decision support systems, yet their evaluation using real patient data across medical specialties and case complexities remains limited. Most existing studies rely on standardized medical examinations rather than authentic clinical scenarios, with only a few using real patient care data. This study systematically evaluates three medical LLMs across multiple medical specialties and case complexity levels using real-world physician assessments. We conducted a prospective cross-sectional evaluation with three board-certified physicians ( n = 3) from internal medicine, surgery, and pediatrics. Each physician categorized 15 real patient cases into three complexity levels (simple, moderate, complex), resulting in a total of 45 cases evaluated ( n = 45). Physicians evaluated LLM performance across three clinical decision support anchors: diagnosis, treatment options, and prescription advice. We assessed MediSearch, MedGemma, and OpenEvidence using a multidimensional framework based on the QUEST methodology, measuring quality, reliability, time efficiency, physician satisfaction, clinical applicability, and safety. Statistical analysis included a three-way ANOVA and post hoc comparisons. MedGemma demonstrated superior overall performance (mean score 7.8 ± 1.2) compared to OpenEvidence (7.1 ± 1.4) and MediSearch (6.4 ± 1.6). All medical LLMs showed significant performance degradation with increasing case complexity (simple: 8.2 ± 0.9; moderate: 7.0 ± 1.1; complex: 6.1 ± 1.3; P < 0.001). Internal medicine cases showed the highest LLM utility (7.6 ± 1.2), followed by pediatrics (7.1 ± 1.4) and surgery (6.6 ± 1.5). Diagnostic accuracy was highest with MedGemma (82%) compared with OpenEvidence (76%) and MediSearch (71%). Critical safety variations emerged, with MedGemma showing significantly lower rates of high-risk recommendations (3.2%) compared to MediSearch (12.4%). In this preliminary evaluation of 45 cases (5 per specialty, evaluated by 3 physicians), medical LLMs demonstrated varying performance across different specialties and case complexity levels. Importantly, all models showed significant performance degradation with increasing case complexity, with the best-performing model (MedGemma) declining by 18% in accuracy from simple to complex cases. The safety assessment identified concerning patterns, including hallucinations, missed contraindications, and dosing errors, across all models, with substantial variation between models. These findings indicate that current medical LLMs have significant limitations that would preclude unmonitored clinical use. This pilot study identifies important knowledge gaps and safety concerns that must be addressed through larger, more rigorous studies before clinical translation can be considered. Future research should prioritize: (1) validation in larger, more diverse case samples across multiple institutions; (2) comprehensive safety assessment in operational settings; (3) investigation of mechanisms underlying performance limitations; and (4) development of appropriate clinical oversight strategies if deployment is eventually pursued. The current evidence does not support clinical implementation recommendations.