Forecasting accuracy is not enough: safety-oriented evaluation of neural digital twins prior to in-silico insulin and carbohydrate intervention in type 1 diabetes

Neural networks trained on a single patient’s continuous glucose monitoring (CGM) record are increasingly proposed as “digital twins” that could let clinicians rehearse insulin and carbohydrate decisions in silico before applying them to the patient. Before any such use, a candidate twin should respond to treatment in a direction consistent with established pharmacology: a model that merely extrapolates the recent glucose trend can report an excellent forecasting error while remaining effectively blind to insulin, and this failure is invisible to the root-mean-square error (RMSE) that dominates the glucose-forecasting literature. We examined this gap in two independent cohorts: a longitudinal, multimodal University of Manchester cohort (17 participants enrolled; 13 formed the Core analytical cohort after data-quality screening) and, as an independent replication of the protocol, the OhioT1DM dataset (12 participants). Patient-specific two-layer long short-term memory (LSTM) models were trained under a strict chronological 60/20/20 split and evaluated across five random seeds. A physiology-guided variant added insulin-on-board and carbohydrate-on-board states with a double-monotonicity training penalty encouraging insulin to lower and carbohydrate to raise predicted glucose. Two findings replicated across both cohorts and all seeds. First, on the accuracy axis, a physiology-free Vanilla LSTM had the lowest one-step RMSE (Manchester 7.00, Ohio 5.02 mg/dL; Friedman χ 2 (5) = 43.4 and 46.7, both p < 10 − 7 ), and under two-hour rollout a non-learning persistence baseline beat every neural model at 60 and 120 minutes—even after rollout-aware training—so ranking twins by accuracy alone would select a model that cannot represent treatment at all. Second, on a graded insulin dose-response audit, the physiology-guided model produced the steepest correctly-directed (glucose-lowering) response among the primary configurations in both cohorts (slope significantly better than the unconstrained model: Holm-corrected p = 0.002 and 0.003), whereas removing the directional constraint left a wrong-signed response; a supplementary ablation without masking was steeper still, pointing to the directional penalty, not masking, as the driver. Its advantage over the simpler Vanilla model was significant on for this cohort dependence. Accuracy and treatment-response plausibility are separable properties; we provide a reproducible protocol auditing both before a neural twin is used for intervention simulation.