123 Large language models in acute neuro-oncology clinics to rapidly produce clinical documentation: are they accurate and do they improve efficiency?

Abstract Introduction Large language models (LLMs) are increasingly used in healthcare to reduce administrative burden and increase productivity. However, the technology’s accuracy in clinical settings is poorly validated. The clinical summaries and letters must be cross-checked by clinicians to verify their accuracy. Acute neuro-oncology clinic patients often have devastating news, which impairs their ability to remember aspects of the consultation beyond the diagnosis. Letters and clinical summaries available immediately after the clinic are helpful for patients and their families, as well as anaesthetic, on-call and ward-based teams. Method Lyrebird LLM was used in 96% of acute oncology clinics, over 6 months, with no patients declining its use. Clinical information was summarised by the LLM and uploaded to the EPR (electronic patient record) to reduce further administrative burden in anaesthetic assessments or perioperative clerking. Letters produced by the LLM, were cross-checked by the oncology fellow for accuracy and compared with the dictated letter to quantify the level of additional detail. Patients and staff completed questionnaires as to the technology’s suitability and utility. Results In 73% of cases, at least 1 change was made to the LLM clinical summary. Frequently, this was due to an incorrect patient’s age, imaging findings phrased according to the patient’s description or a family member’s medical history or experience. However, its use with EPR reduced the burden on clinical teams of repeatedly documenting the patient’s medical history and, as a result, included more details of the presenting complaint. Conclusion The use of LLM becomes more efficient when the clinician is aware of the most common errors and methods to counteract these. Furthermore, it improves the patient’s experience and efficiency of anaesthetic assessments and perioperative clerking.