Abstract / Summary
Abstract Background The spatial organization of cellular lineages within the tumor microenvironment governs disease progression and therapeutic response, but multiplexed spatial profiling remains costly, technically complex, and tissue-destructive. Routine hematoxylin and eosin (H&E) histopathology is inexpensive and widely available but lacks explicit molecular information. This study aims to determine whether single-cell, biomarker-informed cellular states can be inferred directly from H&E images to enable scalable, non-destructive spatial biology across cancer types and clinical endpoints. Methods We present CellPlexer, a deep learning framework trained on more than 36 million co-registered cells from paired H&E and multiplex immunofluorescence images to jointly segment nuclei and classify each cell into one of 15 hierarchically defined biomarker states. A spatially aware transformer integrates single-cell embeddings with patch-level histomorphology through attention-based multimodal fusion, generating a multi-scale whole-slide representation for clinical prediction. We validated CellPlexer across four independent cohorts spanning lung, breast, and colorectal cancer. Results Here we show that CellPlexer consistently outperformed morphology-only baselines across recurrence, treatment response, survival, and mutation prediction tasks, achieving an AUC of 72.19% for pathological complete response prediction in triple-negative breast cancer (75.06% with clinical variable integration) and 83.09% for BRAF mutation prediction in colorectal cancer. Conclusions These results demonstrate that morphologically inferred, single-cell biomarker states capture clinically and biologically meaningful tumor microenvironment organization, providing a scalable, non-destructive framework for extracting proteomic-level insights from widely available clinical histopathology archives.