Quality and Safety of ChatGPT Responses to Anesthesia-Related Patient Questions: An Expert Evaluation

AbstractAim: Large language models such as ChatGPT are increasingly used by patients to obtain perioperative information. Given the complexity and potential risks associated with anesthetic management, the accuracy and safety of such information are of critical importance. This study aimed to evaluate the quality and potential clinical risks of ChatGPT-generated responses to anesthesia-related questions using a structured scoring system.Materials and Methods: A total of 100 anesthesia-related questions were evaluated, including 50 guideline-derived questions and 50 frequently asked public questions. All questions were submitted to the ChatGPT-4 model in Turkish using the premium version. Responses were independently evaluated by three anesthesiology specialists using the Global Quality Score (GQS; 1–5). In addition, each response was assessed for the presence of potentially harmful or misleading information. Comparisons were conducted between question categories.Results: The median Global Quality Score (GQS) differed significantly between question types, with higher scores observed for frequently asked questions than for guideline-based questions (4 [IQR: 3–5] vs. 3 [IQR: 2–4], p < 0.001). Potentially harmful or misleading information was identified in 7% of responses and was more frequent in guideline-based questions than in frequently asked questions (12% vs. 2%, p = 0.02).Conclusion: ChatGPT provides high-quality and comprehensible responses to general anesthesia-related questions. However, its performance declines in guideline-based, clinically critical topics. Although the overall risk rate is relatively low, the presence of potentially misleading content underscores the need for clinician oversight when such tools are used for patient education.Keywords: ChatGPT; anesthesiology; artificial intelligence; patient education; global quality score