Multimodal artificial intelligence assessment of simulated cardiac arrest performance: a pilot study

Objectives Assessment of simulated cardiac arrest performance is resource-intensive and subject to inter-rater variability. We evaluated whether a multimodal artificial intelligence (AI) system could score video-recorded cardiac arrest megacode simulations with agreement comparable to expert raters. Methods Six scripted megacode simulations (three ventricular fibrillation, three pulseless electrical activity) representing exemplar, guideline-deviation and non-technical-deficiency performance were scored by two ACLS instructors using a validated team-leader checklist; consensus ratings served as the reference. Two operators analysed each video with Gemini 1.5 Pro at three temperature settings (504 item-level ratings). We calculated Cohen’s kappa, Gwet’s AC1, and sensitivity and specificity for identifying performed and omitted actions, and compared AI self-reported confidence between correct and incorrect ratings. Results Expert inter-rater agreement was κ = 0.78 (95% CI 0.64–0.93). AI–expert agreement was κ = 0.62 (95% CI 0.49–0.75; range 0.40–0.86 across configurations) and AC1 = 0.77. The AI identified performed actions with high sensitivity (0.98, 95% CI 0.96–0.99) but detected omitted actions poorly (specificity 0.58, 95% CI 0.44–0.72), particularly omissions defined by timing or sequence. Temperature had no consistent effect. Incorrect ratings of actions as performed carried lower mean self-reported confidence than correct ones (93.6 vs 98.1), but 48% were made with maximal confidence. Conclusions A multimodal AI system reliably confirmed completed actions but frequently failed to detect omitted actions, yielding moderate overall agreement that was below expert–expert agreement. Current systems may support, but not replace, expert assessment of cardiac arrest simulation.