Evaluating large language models for selected public health research and practice tasks: A 100-question benchmark of reliability, workflow-assisted recoverability, and correct source-grounded abstention

Background Large language models (LLMs) are increasingly used in public health, but performance on general-purpose benchmarks may not translate directly to routine, evidence-grounded public health tasks. Objective To evaluate LLM performance on a selected sample of public health tasks, emphasizing first-attempt reliability, workflow-assisted correction, and correct source-grounded abstention. Methods We evaluated 100 questions across literature-based factual extraction and summarization (n=56), vaccine-specific retrieval and absence detection (n=10), study design and sample-size calculation (n=27), and statistical analysis (n=7). Experiments were conducted during December 2024 to April 2025 with some rechecks performed in May-June 2026. Initial attempts generally used GPT-4o in public web interfaces; recognized failures were repeated with the same model or escalated to a more capable model. Responses were manually checked and graded against verified reference standards. Results First-attempt accuracy was 72/100 (72.0%; 95% Wilson CI, 62.5%-79.9%): literature 53/56 (94.6%; 85.4%-98.2%), vaccine-specific retrieval 3/10 (30.0%; 10.8%-60.3%), study design 12/27 (44.4%; 27.6%-62.7%), and statistical analysis 4/7 (57%; 25%-84%). Among ten questions that required source-grounded abstention, 2/10 (20.0%; 5.7%-51.0%) were correct initially, including 0/7 (0%; 0%-35%) vaccine-specific items. All initial abstention failures were later corrected after repeat/escalation. Because retry/escalation continued after recognized failures, eventual correctness of 100% reflected the stopping rule rather than autonomous model accuracy. Conclusions LLMs can assist selected public health tasks, but first-attempt reliability, task-specific validation, and source-grounded abstention remain critical for real-world use.