Abstract / Summary
Living clinical guidelines revise recommendations as evidence emerges, so an AI answer can be faithful to a real guideline and still be out of date. Here we examine this failure using ASCOBench, 288 unique questions in 96 three-turn conversations grounded in versioned American Society of Clinical Oncology (ASCO) breast and prostate cancer guidelines, with oncologist-reviewed reference answers. Oncologists corrected one in four version-sensitive answers drafted by a frontier model, against fewer than one in ten factual ones. Across the guideline corpus, 19 recommendations changed between versions, seven of them reversals, yet only 1 of 14 superseded documents states that it has been replaced. With a strong model, retrieval over several guideline versions made stale answers four times more frequent than answering without retrieval. SentryLine, a verification-first system, lowered incorrect answers to 4.2-5.2% from 9.7-22.9% for baselines across three models. Recognizing change before a guideline declares it remains an open problem.