Abstract / Summary
Abstract Purpose To compare the diagnostic and treatment performance of DeepSeek-R1 and GPT-4o large language models (LLMs) in ophthalmology using standardized residency examination cases.
Design: Cross-sectional comparative study. Participants: Thirty-three representative cases drawn from the Chinese Ophthalmology Residency Examination Database, covering 8 subspecialties. Methods Each case was processed by DeepSeek-R1 and GPT-4o with identical prompts to act as senior ophthalmologists. Three independent ophthalmologists conducted double-blind evaluations of each model’s outputs. Accuracy was scored on a 10-point Likert scale and completeness on a 6-point Likert scale for diagnosis, differential diagnosis, and treatment. Mean scores were compared using paired statistical tests and two-way ANOVA. Main Outcome Measures: Accuracy and completeness scores across diagnosis, differential diagnosis, and treatment tasks. Results Across all cases, DeepSeek-R1 achieved significantly higher accuracy for diagnosis (8.04 vs 6.46, P < 0.0001), differential diagnosis (7.52 vs 5.50, P < 0.0001), and treatment (7.62 vs 6.65, P = 0.002) compared with GPT-4o. Completeness scores were also superior for DeepSeek-R1 in diagnosis (4.86 vs 3.69, P < 0.0001), differential diagnosis (4.44 vs 3.24, P < 0.0001), and treatment (4.61 vs 3.90, P = 0.0001). Subspecialty analyses revealed the largest advantage for DeepSeek-R1 in retinal diseases, glaucoma, strabismus & amblyopia, and optic nerve disorders. Conclusions In this time-stamped, text-only evaluation of standardized ophthalmology cases conducted between February and March 2025, DeepSeek-R1 achieved higher overall mean accuracy and completeness scores than GPT-4o. However, not all subgroup comparisons reached statistical significance, and the findings should be interpreted as preliminary given the limited sample size and text-only design.