Abstract / Summary
Social media language and digital lifestyle behaviours are promoted as scalable signals of population mental health, but predictive models are often evaluated in ways that overstate accuracy. We analysed 20,000 tweets from 72 users, labelled by whether the account belonged to a user identified as depressed, and two synthetic datasets (100,000 and 30,000 records) linking screen time, sleep and social media use with self-reported stress. Under a conventional tweet-level split, a term-weighted logistic regression appeared accurate (area under the receiver operating characteristic curve 0.835), but when whole users were held out its performance fell to near chance (0.595; 95% confidence interval 0.505 to 0.686), because the model largely recognised users and their topics. Depression-associated tweets were more often negative (31.7% versus 24.2%), yet sentiment alone could not identify unseen users (0.539). The two behavioural datasets gave opposite answers: strong discrimination in one (0.926), driven by short sleep, and chance in the other (0.506), so synthetic results reflect how the data were generated. At a population prevalence of 10%, positive predictive values would be 0.10 to 0.30. Digital distress signals should be validated across individuals, on real linked data and at realistic prevalence before informing public health surveillance or prevention.