Comparative assessment of patient education content generated by five large language models on post-stroke complications: A cross-sectional study of safety, accuracy, reliability, readability, and empathy

Objective To compare the safety, accuracy, reliability, readability, and empathy of patient education content generated by five large language models (LLMs) for post-stroke complications. Methods This CHART-guided cross-sectional comparative study evaluated ChatGPT, Gemini, DeepSeek, Doubao, and Qwen using 40 standardized patient- and caregiver-facing prompts. Each model generated one response per prompt in an independent session. Three trained raters—a neurologist and two nursing postgraduate researchers specializing in neurological and stroke rehabilitation nursing—independently assessed safety, accuracy, empathy, and information quality using predefined criteria, including DISCERN, EQIP, GQS, and JAMA. Readability was assessed using six established indices. Agreement was evaluated using Fleiss’ kappa and ICC(2,1). Model comparisons used Cochran’s Q test for safety and Friedman tests with Kendall’s W for non-binary outcomes, followed by Benjamini–Hochberg-adjusted pairwise analyses where appropriate. Results The final dataset comprised 2,600 metric-level observations. Inter-rater agreement was acceptable to high (Fleiss’ kappa = 0.740; ICCs = 0.838–0.893). Safe-response rates were high (95.0%–97.5%) and did not differ significantly ( Q = 0.750, P = 0.945). Significant between-model differences were found for accuracy, empathy, DISCERN, EQIP, GQS, and all readability indices (all P < 0.001), although the overall effect for accuracy was modest ( W = 0.230). JAMA scores were uniformly low and showed no significant difference ( P = 0.888), indicating limited transparency in standalone chatbot responses. No model consistently met the recommended sixth-grade reading level or achieved mean Flesch Reading Ease ≥80. Conclusions The evaluated LLMs generated mostly safe and clinically relevant English-language responses, but performance varied across domains and no model was consistently superior. LLMs may assist in drafting patient education materials, but clinical review, evidence verification, source attribution, and readability optimization remain necessary before patient-facing use. Findings are specific to the prompts and single-generation outputs evaluated.