Development and Internal Validation of a Safety-Gated Artificial Intelligence System for Predicting Serious Bacterial Infection among Children in Northern Nigeria

Abstract Introduction: Serious bacterial infection (SBI), remain a major cause of morbidity and mortality among children under five, particularly in low- and middle-income countries where diagnostic capacity is limited. Despite increasing use of artificial intelligence (AI) in pediatric risk prediction, context-specific AI models for pediatric SBI in Nigeria remain limited. To develop and internally validate an AI-based clinical prediction framework for identifying SBI risk among children in Nigerian hospital settings, with additional safety-gating features to support clinically responsible use. Methods We conducted an AI-enabled retrospective prediction model development and internal validation using pediatric clinical records from Nigerian hospital settings. Ensemble prediction combined logistic regression, decision tree, and Extreme Gradient Boosting models implemented in R. Internal validation used a stratified 80:20 development-held-out test split, with tuning, class weighting, and threshold optimization performed within the development partition only. The system architecture incorporated a local large language model, by finetuning MedGemma, for schema-constrained extraction from free-text clinical-note inputs, and a Python-based deterministic gating engine for completeness, plausibility, consistency, and prediction-readiness checks. Metrics including AUROC, sensitivity, specificity, accuracy, predictive values, balanced accuracy, and simulated workflow-abstention rates. Results Among 11,466 labeled pediatric records, 936 (8.2%) were SBI-positive. Culture-confirmed SBI was associated with key demographic, anthropometric, clinical, physiologic, and comorbidity factors, while the strongest model predictors were mainly growth, age-related, and maternal factors, with selected clinical signs contributing additional predictive value. The finalized ensemble model achieved AUROC 0.76 (95% CI: 0.72–0.80), sensitivity 0.76 (95% CI: 0.69–0.82), specificity 0.63 (95% CI: 0.61–0.65), accuracy 0.64 (95% CI: 0.62–0.66), PPV 0.13 (95% CI: 0.11–0.16), NPV 0.97 (95% CI: 0.96–0.98), and balanced accuracy 0.70 (95% CI: 0.66–0.73), using an optimized threshold of 0.50. In simulated workflow testing, schema-conforming extraction succeeded in 95% of inputs, while prediction was withheld in 20% because of missing, implausible, or conflicting information. Conclusion This study presents a context-specific AI framework for pediatric SBI prediction using Nigerian clinical data. The findings support further external validation, prospective evaluation, and testing with real-world clinical documentation before routine deployment.