Validation-set-constrained three-path prediction-score fusion for in-hospital mortality at 24 hours after ICU admission: internal evaluation and a cross-database transportability stress test

Abstract Background Structured physiological time series and clinical notes capture different aspects of illness in the intensive care unit (ICU), yet multimodal models may lose their advantage when inputs are unavailable or data environments change. We developed a validation-set-constrained three-path prediction-score fusion framework with a multimodal primary path and independently trained notes-only and physiology paths, and evaluated its incremental value, fallback performance, and transportability. Methods We used a 24-hour landmark design and included 33,484 ICU stays from 27,667 adults in MIMIC-III. The paths were trained independently using patient-grouped five-fold cross-validation, and their outputs were combined with non-negative convex weights selected solely by inner-validation area under the precision–recall curve (AUPRC). Analyses included framework-level path ablation, modality interruption, the CareVue-to-MetaVision transition, and eICU transfer analyses of fixed source weights, exploratory reweighting, text quality, and cross-database representations. Results In internal MIMIC-III evaluation, full three-path fusion achieved an area under the receiver operating characteristic curve (AUROC) of 0.8887 and an AUPRC of 0.5508, exceeding the multimodal model by 0.0069 (95% confidence interval [CI] 0.0056–0.0082) and 0.0122 (95% CI 0.0062–0.0183), respectively. Fusion remained superior after the CareVue-to-MetaVision transition (0.7639/0.3234 vs 0.7452/0.2996), but under direct transfer to eICU it underperformed the physiology path (0.6594/0.2160 vs 0.7327/0.2840). The source-domain fusion assigned a mean 71.4% weight to the multimodal path. Exploratory target-domain reweighting assigned a mean 93.0% weight to the physiology path and yielded held-out AUROC/AUPRC of 0.7332/0.2839. Platt recalibration reduced expected calibration error (ECE) from 0.3214 to 0.0132 without changing discrimination. Conclusions Validation-set-constrained fusion provided a small but consistent internal AUPRC gain and prespecified fallback rules for unavailable modalities. The gain persisted across the CareVue-to-MetaVision EHR-system transition but not in eICU, where path ranking reversed and source weights continued to emphasize the degraded multimodal path. Before use in a new environment, input correspondence, path performance, fusion weights, and calibration require reassessment.