Correct Answers With Invalid Explanations by a Multimodal Large Language Model on a Public Ophthalmology Examination

Objective To estimate invalid user-facing explanations among correct GPT-5.4 Pro answers and assess sensitivity to adjudication rule, item clustering, and audit sampling. Design Cross-sectional repeated-trial model evaluation with specialist explanation audit. Subjects, Participants, and/or Controls Model-generated responses to the 2023 Japanese Ophthalmological Society specialty examination; no human participants or controls. A stratified sample of 370 GPT-5.4 Pro responses was audited. Methods, Intervention, or Testing Nine ChatGPT releases completed 20 examination trials in Japanese and translated English. Three ophthalmologists independently rated GPT-5.4 Pro explanations and resolved disagreements by discussion. Sensitivity analyses used any-rater, majority-vote, and unanimous rules. Accuracy and invalidity confidence intervals used bootstrap resampling clustered by examination item; pooled estimates were weighted for item-specific sampling probabilities. Results GPT-5.4 Pro achieved mean Japanese scores of 99.9/100 for general and 92.7/100 for clinical questions. Audited-sample accuracy was 353/370 (95.4%; item-cluster 95% CI, 91.0%-98.9%); the design-weighted estimate was 97.3% (95% CI, 94.8%-99.3%). Among correct responses, consensus classified 52/353 explanations as invalid (14.7%; 95% CI, 10.1%-19.8%), whereas majority vote classified 18/353 (5.1%; 95% CI, 2.6%-8.1%). Design-weighted pooled estimates were 14.6% (95% CI, 9.9%-19.6%) and 2.9% (95% CI, 1.5%-4.6%), respectively. Consensus estimates were 13.3% for general and 15.8% for clinical correct responses. No general response met majority vote. Exact 3-rater agreement was 68.6% and Fleiss kappa was 0.234. Conclusions On this public benchmark, correct answers did not ensure valid explanations. The difference between consensus and majority estimates shows material adjudication-threshold dependence. The findings concern user-facing explanations, not internal reasoning traces, and do not determine whether correct answers resulted from image interpretation, retrieval, or prior exposure.