Development and temporal validation of a calibrated machine learning model for in-hospital blood culture positivity using routine clinical data

Background Among hospitalized patients who have already undergone blood culture collection, routinely available clinical information may support early risk stratification while microbiological results are pending. We developed and temporally validated a calibrated machine learning model for in-hospital blood culture positivity. Methods This single-center retrospective study included 44,389 blood culture-related hospitalization records from January 2022 to September 2024. Records from 2022 to 2023 were used for model development and records from 2024 for temporal validation. The primary outcome was at least one laboratory-recorded positive blood culture between index collection and hospital discharge. A Platt-calibrated XGBoost model using demographic characteristics, routine laboratory variables, and missingness indicators was compared with procalcitonin-only and regression-based models. Performance was assessed using discrimination, calibration, threshold-specific metrics, decision curve analysis, and SHAP. Results The cohort included 3,555 outcome-positive records (8.01%). In temporal validation, the final XGBoost model achieved a ROC-AUC of 0.798 (95% CI, 0.784–0.813), a PR-AUC of 0.266, a Brier score of 0.0619, and a calibration slope of 1.061. At the 5% threshold, sensitivity was 0.877, specificity 0.570, and negative predictive value 0.983. XGBoost showed a modest improvement over logistic regression with missingness indicators, while an exploratory post hoc Age + PCT + BMI logistic comparator achieved a ROC-AUC of 0.757. Performance was stable in the new-patient validation subset but decreased within adult and pediatric subgroups (ROC-AUC 0.679 and 0.663, respectively). Age, procalcitonin, and body mass index were the dominant contributors. Conclusion The calibrated model provided moderate discrimination and favorable calibration for hospitalization-level blood culture positivity risk stratification in the pooled temporal validation cohort. Performance varied across age groups, and much of the predictive information was concentrated in age, PCT, and BMI. The model does not predict the index culture result alone or diagnose clinically adjudicated bloodstream infection. Multicenter external and prospective validation using fixed prediction horizons and adjudicated outcomes is required before clinical implementation.