Evaluation of large language models for ICI-related liver injury with an emerging microbiome focus

Background Immune checkpoint inhibitor-related liver injury (ICI-LI) has heterogeneous phenotypes, a complex differential diagnosis, and evolving management. Gut microbiome research provides biological plausibility for immune regulation during ICI therapy, but evidence specific to ICI-LI remains limited and is not yet clinically actionable. We evaluated whether publicly accessible large language model (LLM) chatbots can organize this mixed evidence base and communicate its uncertainty. Methods We developed a disease-specific benchmark of 25 questions covering immunological mechanisms, microbiome and immune regulation, diagnosis, biomarkers, and management of ICI-LI. The benchmark addressed ICI-LI broadly and included a focused microbiome component. Four publicly available LLM platforms were evaluated using DISCERN, EQIP, the Global Quality Score, the JAMA Benchmark Criteria, and six readability indices. These instruments assessed information quality, structure, transparency, and patient-facing readability rather than factual accuracy or clinical decision-making. Results Performance differed across DISCERN, EQIP, and JAMA scores, whereas the Global Quality Score showed limited discrimination. Question-level variation was most apparent for items requiring evidence appraisal or translation. All platforms produced structured responses, but none met the prespecified patient-facing readability benchmark. Readability was interpreted separately from clinician-facing information synthesis. Conclusions Disease-specific multidimensional evaluation is needed before LLM-generated information can be considered reliable for microbiome-informed clinical communication or decision support. Evaluation should emphasize evidence calibration and distinguish biological plausibility, association, and clinically actionable evidence.