How capable are large language models in specialized medical domains? The case of Hashimoto’s thyroiditis

To systematically assess the accuracy, completeness, and safety of information provided by three mainstream LLMs (ChatGPT-5, DeepSeek-V3.2, Gemini-2.5 Flash) regarding Hashimoto’s Thyroiditis (HT), a set of standardized clinical questions was designed and posed to the LLMs (Foundational Knowledge, Clinical Presentation and Diagnosis, Treatment and Management). Two independent endocrinologists blinded to the model sources evaluated each response. Using authoritative references from the National Institutes of Health (NIH) and the American Thyroid Association (ATA) as the gold standard, they scored each answer on factual accuracy, completeness, medical safety, and source support. The weighted total scores and rankings of the three LLMs across all metrics were calculated: Gemini-2.5 Flash (93.5) > DeepSeek-V3.2 (88) > ChatGPT-5 (85). Comparison of normalized item scores showed that Gemini-2.5 Flash scored significantly higher than both other models, whereas DeepSeek-V3.2 and ChatGPT-5 did not differ significantly. Qualitative analysis identified common strengths (e.g., chain-of-thought process, proactive engagement, generally accurate and comprehensive responses) and weaknesses (e.g., outdated or non-existent information, omission of key points, absolute claims lacking sufficient evidence). Specific LLMs demonstrate potential in providing information on HT; however, the presence of inconsistencies and risks necessitates their cautious use.