Large language model-assisted machine learning for postpartum urinary incontinence prediction: Model development and validation study

Objectives Accurate prediction of postpartum urinary incontinence (UI) remains challenging, as existing models often fail to fully utilize information in unstructured electronic medical records (EMRs). This study aimed to develop and validate a prediction model integrating structured and unstructured clinical data to improve postpartum UI risk assessment. Methods Two prospective cohorts were recruited for model development and validation. A large language model (LLM) was used to extract clinical information from unstructured EMRs and integrate it with structured data. Seven machine learning algorithms were trained and internally validated using stratified five-fold cross-validation, with the optimal model selected based on the area under the curve (AUC). The final model was externally validated and compared with a baseline logistic regression model developed using structured data only. Shapley Additive Explanations (SHAP) was used to interpret feature importance. Results A total of 1605 pregnant women were included, and 20.7% experienced postpartum UI. LLM achieved up to 98.9% overall accuracy in extracting information from unstructured EMRs. Among the seven models, the multilayer perceptron (MLP) achieved the best performance, with internal validation and external validation AUC of 0.813 and 0.782, respectively, outperforming the baseline model. SHAP identified severity of UI during pregnancy, duration of second stage of labor, and number of vaginal deliveries as the most influential predictive factors in the model. A web-based calculator was developed to support future clinical implementation. Conclusions By incorporating a LLM into the development process, we established a robust prediction model for postpartum UI. This innovative approach may facilitate early risk identification.