How Robust Are Machine-Learning Biomarkers of Depression? A Leakage-Controlled Benchmark

Abstract Background Machine-learning models are increasingly used to identify predictive markers of depression, yet reported performance may be sensitive to data leakage, non-independent observations, site effects, and validation procedures that do not adequately approximate deployment in independent clinical populations. Moreover, increasingly complex algorithms may provide limited incremental value over simpler statistical models once leakage is controlled. This study examined how validation design and leakage-prone predictors influence estimates of depression-classification performance and whether complex machine-learning models outperform logistic regression under more rigorous evaluation conditions. Methods A controlled synthetic multisite psychiatric dataset comprising 13,159 observations from 8,000 unique synthetic participants across 10 simulated sites was used to benchmark five classification approaches: logistic regression, random forest, extreme gradient boosting (XGBoost), support vector machine, and a neural network. The dataset incorporated repeated observations, demographic characteristics, depression- and anxiety-related symptom measures, sleep and activity indicators, physiological measures, cognitive performance, treatment variables, site and device effects, and realistic missingness. Depression case status was defined using a synthetic PHQ-9-like total score threshold of ≥ 10, yielding a prevalence of 35.4%. To prevent direct target leakage, the PHQ-9-like total score was excluded from predictive models. Two deliberately post-outcome, leakage-prone variables were additionally manipulated to quantify performance inflation. Leakage-controlled models were evaluated using conventional random row splitting, participant-independent splitting, and site-independent validation. Discrimination and probabilistic performance were assessed using area under the receiver operating characteristic curve (AUROC), area under the precision-recall curve (AUPRC), and Brier score. Results Allowing leakage-prone post-outcome predictors produced markedly optimistic performance, with random-split AUROCs of approximately 0.988–0.989 and AUPRCs of 0.981–0.983. For logistic regression, removal of leakage-prone predictors reduced AUROC from 0.9891 to 0.9548 and AUPRC from 0.9827 to 0.9253, while the Brier score increased from 0.0367 to 0.0788. In leakage-controlled analyses, discrimination nevertheless remained high across validation strategies. AUROCs ranged from 0.9488 to 0.9548 under random row splitting, 0.9516 to 0.9585 under participant-independent validation, and 0.9502 to 0.9555 under site-independent validation. Corresponding AUPRCs ranged from 0.9116 to 0.9254, 0.9246 to 0.9383, and 0.9183 to 0.9356, respectively. Logistic regression achieved an AUROC of 0.9585 and AUPRC of 0.9383 under participant-independent validation and performed comparably to random forest, XGBoost, support vector machine, and neural-network classifiers. Increasing algorithmic complexity therefore yielded little improvement in discrimination after leakage control. Conclusions Apparent performance in psychiatric machine-learning benchmarks can be substantially influenced by leakage-prone predictors, even when underlying predictors contain strong signal. In this controlled synthetic benchmark, removing post-outcome information materially reduced apparent discrimination, whereas participant- and site-independent validation did not produce a further collapse in performance. Complex machine-learning algorithms provided little predictive advantage over logistic regression. These findings illustrate why leakage auditing, explicit separation of outcome-derived variables, clinically realistic validation, calibration assessment, and comparison against simple baselines should be routine components of psychiatric machine-learning research. Because the present findings derive from synthetic data, they demonstrate methodological mechanisms rather than establishing the magnitude of performance inflation in real clinical populations; replication using independent empirical psychiatric datasets is required.